Intensity transform augmentation system and methods for use therewith

ABSTRACT

An intensity transform augmentation system is operable to generate a plurality of sets of augmented images by performing a set of intensity transformation functions on each of a training set of medical scans. Each of the set of intensity transformation functions are based on density properties of corresponding anatomy feature present in the training set of medical scans. A computer vision model is generated by performing a training step on the plurality of sets of augmented images, where each augmented image of a set of augmented images is assigned same output label data based on a corresponding one of the training set of medical scans. Inference data is generated by performing an inference function on a new medical scan by utilizing the computer vision model on the new medical scan. The inference data is transmitted to a client device for display via a display device.

CROSS REFERENCE TO RELATED APPLICATIONS

The present U.S. Utility patent application claims priority pursuant to35 U.S.C. § 120 as a continuation of U.S. Utility application Ser. No.16/360,275, entitled “UTILIZING DENSITY PROPERTIES OF ANATOMICALFEATURES IN AN INTENSITY TRANSFORM AUGMENTATION SYSTEM”, filed Mar. 21,2019, which claims priority pursuant to 35 U.S.C. § 119(e) to U.S.Provisional Application No. 62/770,334, entitled “LESION TRACKINGSYSTEM”, filed Nov. 21, 2018, both of which are hereby incorporatedherein by reference in their entirety and made part of the present U.S.Utility patent application for all purposes.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not applicable.

INCORPORATION-BY-REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISC

Not applicable.

BACKGROUND Technical Field

This invention relates generally to medical imaging devices andknowledge-based systems used in conjunction with client/server networkarchitectures.

Description of Related Art BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THEDRAWING(S)

FIG. 1 is a schematic block diagram of an embodiment;

FIG. 2A is a schematic block diagram of a client device in accordancewith various embodiments;

FIG. 2B is a schematic block diagram of one or more subsystems inaccordance with various embodiments;

FIG. 3 is a schematic block diagram of a database storage system inaccordance with various embodiments;

FIG. 4A is schematic block diagram of a medical scan entry in accordancewith various embodiments;

FIG. 4B is a schematic block diagram of abnormality data in accordancewith various embodiments;

FIG. 5A is a schematic block diagram of a user profile entry inaccordance with various embodiments;

FIG. 5B is a schematic block diagram of a medical scan analysis functionentry in accordance with various embodiments;

FIGS. 6A-6B are schematic block diagram of a medical scan diagnosingsystem in accordance with various embodiments;

FIG. 7A is a flowchart representation of an inference step in accordancewith various embodiments;

FIG. 7B is a flowchart representation of a detection step in accordancewith various embodiments;

FIGS. 8A-8F are schematic block diagrams of a medical picture archiveintegration system in accordance with various embodiments;

FIG. 9 is a flowchart representation of a method for execution by amedical picture archive integration system in accordance with variousembodiments;

FIG. 10A is a schematic block diagram of a de-identification system inaccordance with various embodiments;

FIG. 10B is an illustration of an example of anonymizing patientidentifiers in image data of a medical scan in accordance with variousembodiments;

FIG. 11 presents a flowchart illustrating a method for execution by ade-identification system in accordance with various embodiments;

FIG. 12A is a schematic block diagram of an intensity transformaugmentation system in accordance with various embodiments;

FIG. 12B illustrates an example of a pre-processing step utilized by anintensity transform augmentation system in accordance with variousembodiments;

FIGS. 12C-12F illustrate example embodiments of intensity transformationfunctions utilized by an intensity transform augmentation system inaccordance with various embodiments;

FIG. 12G is a schematic block diagram of an intensity transformaugmentation system in accordance with various embodiments;

FIG. 12H is a schematic block diagram of an intensity transformaugmentation system in accordance with various embodiments;

FIG. 13 presents a flowchart illustrating a method for execution by anintensity transform augmentation system in accordance with variousembodiments;

FIG. 14 presents a flowchart illustrating a method for execution by anintensity transform augmentation system in accordance with variousembodiments; and

FIG. 15 presents a flowchart illustrating a method for execution by acontrast parameter learning system in accordance with variousembodiments.

DETAILED DESCRIPTION

The present U.S. Utility patent application is related to U.S. Utilityapplication Ser. No. 15/627,644, entitled “MEDICAL SCAN ASSISTED REVIEWSYSTEM”, filed 20 Jun. 2017, which claims priority pursuant to 35 U.S.C.§ 119(e) to U.S. Provisional Application No. 62/511,150, entitled“MEDICAL SCAN ASSISTED REVIEW SYSTEM AND METHODS”, filed 25 May 2017,both of which are hereby incorporated herein by reference in theirentirety and made part of the present U.S. Utility patent applicationfor all purposes.

FIG. 1 presents a medical scan processing system 100, which can includeone or more medical scan subsystems 101 that communicate bidirectionallywith one or more client devices 120 via a wired and/or wireless network150. The medical scan subsystems 101 can include a medical scan assistedreview system 102, medical scan report labeling system 104, a medicalscan annotator system 106, a medical scan diagnosing system 108, amedical scan interface feature evaluator system 110, a medical scanimage analysis system 112, a medical scan natural language analysissystem 114, and/or a medical scan comparison system 116. Some or all ofthe subsystems 101 can utilize the same processing devices, memorydevices, and/or network interfaces, for example, running on a same setof shared servers connected to network 150. Alternatively or inaddition, some or all of the subsystems 101 be assigned their ownprocessing devices, memory devices, and/or network interfaces, forexample, running separately on different sets of servers connected tonetwork 150. Some or all of the subsystems 101 can interact directlywith each other, for example, where one subsystem's output istransmitted directly as input to another subsystem via network 150.Network 150 can include one or more wireless and/or wired communicationsystems; one or more non-public intranet systems and/or public internetsystems; and/or one or more local area networks (LAN) and/or wide areanetworks (WAN).

The medical scan processing system 100 can further include a databasestorage system 140, which can include one or more servers, one or morememory devices of one or more subsystems 101, and/or one or more othermemory devices connected to network 150. The database storage system 140can store one or more shared databases and/or one or more files storedon one or more memory devices that include database entries as describedherein. The shared databases and/or files can each be utilized by someor all of the subsystems of the medical scan processing system, allowingsome or all of the subsystems and/or client devices to retrieve, edit,add, or delete entries to the one or more databases and/or files.

The one or more client devices 120 can each be associated with one ormore users of one or more subsystems of the medical scan processingsystem. Some or all of the client devices can be associated withhospitals or other medical institutions and/or associated with medicalprofessionals, employees, or other individual users for example, locatedat one or more of the medical institutions. Some of the client devices120 can correspond to one or more administrators of one or moresubsystems of the medical scan processing system, allowingadministrators to manage, supervise, or override functions of one ormore subsystems for which they are responsible.

Some or all of the subsystems 101 of the medical scan processing system100 can include a server that presents a website for operation via abrowser of client devices 120. Alternatively or in addition, each clientdevice can store application data corresponding to some or allsubsystems, for example, a subset of the subsystems that are relevant tothe user in a memory of the client device, and a processor of the clientdevice can display the interactive interface based on instructions inthe interface data stored in memory. For example, the website presentedby a subsystem can operate via the application. Some or all of thewebsites presented can correspond to multiple subsystems, for example,where the multiple subsystems share the server presenting the website.Furthermore, the network 150 can be configured for secure and/orauthenticated communications between the medical scan subsystems 101,the client devices 120 and the database storage system 140 to protectthe data stored in the database storage system and the data communicatedbetween the medical scan subsystems 101, the client devices 120 and thedatabase storage system 140 from unauthorized access.

The medical scan assisted review system 102 can be used to aid medicalprofessionals or other users in diagnosing, triaging, classifying,ranking, and/or otherwise reviewing medical scans by presenting amedical scan for review by a user by transmitting medical scan data of aselected medical scan and/or interface feature data of selectedinterface features of to a client device 120 corresponding to a user ofthe medical scan assisted review system for display via a display deviceof the client device. The medical scan assisted review system 102 cangenerate scan review data for a medical scan based on user input to theinteractive interface displayed by the display device in response toprompts to provide the scan review data, for example, where the promptscorrespond to one or more interface features.

The medical scan assisted review system 102 can be operable to receive,via a network, a medical scan for review. Abnormality annotation datacan be generated by identifying one or more of abnormalities in themedical scan by utilizing a computer vision model that is trained on aplurality of training medical scans. The abnormality annotation data caninclude location data and classification data for each of the pluralityof abnormalities and/or data that facilitates the visualization of theabnormalities in the scan image data. Report data including textdescribing each of the plurality of abnormalities is generated based onthe abnormality data. The visualization and the report data, which cancollectively be displayed annotation data, can be transmitted to aclient device. A display device associated with the client device candisplay the visualization in conjunction with the medical scan via aninteractive interface, and the display device can further display thereport data via the interactive interface.

In various embodiments, longitudinal data, such as one or moreadditional scans of longitudinal data 433 of the medical scan or ofsimilar scans, can be displayed in conjunction with the medical scanautomatically, or in response to the user electing to view longitudinaldata via user input. For example, the medical scan assisted reviewsystem can retrieve a previous scan or a future scan for the patientfrom a patient database or from the medical scan database automaticallyor in response to the user electing to view past patient data. One ormore previous scans can be displayed in one or more correspondingwindows adjacent to the current medical scan. For example, the user canselect a past scan from the longitudinal data for display. Alternativelyor in addition, the user can elect longitudinal parameters such asamount of time elapsed, scan type, electing to select the most recentand/or least recent scan, electing to select a future scan, electing toselect a scan at a date closest to the scan, or other criteria, and themedical scan assisted review system can automatically select a previousscan that compares most favorably to the longitudinal parameters. Theselected additional scan can be displayed in an adjacent windowalongside the current medical scan. In some embodiments, multipleadditional scans will be selected and can be displayed in multipleadjacent windows.

In various embodiments, a first window displaying an image slice 412 ofthe medical scan and an adjacent second window displaying an image sliceof a selected additional scan will display image slices 412 determinedto correspond with the currently displayed slice 412 of the medicalscan. As described with respect to selecting a slice of a selectedsimilar medical scan for display, this can be achieved based onselecting the image slice with a matching slice number, based onautomatically determining the image slice that most closely matches theanatomical region corresponding to the currently displayed slice of thecurrent scan, and/or based on determining the slice in the previous scanwith the most similar view of the abnormality as the currently displayedslice. The user can use a single scroll bar or other single user inputindication to jump to a different image slice, and the multiple windowscan simultaneously display the same numbered image slice, or can scrollor jump by the same number of slices if different slice numbers areinitially displayed. In some embodiments, three or more adjacent windowscorresponding to the medical scan and two or more additional scans aredisplayed, and can all be controlled with the single scroll bar in asimilar fashion.

The medical scan assisted review system 102 can automatically detectprevious states of the identified abnormalities based on the abnormalitydata, such as the abnormality location data. The detected previousstates of the identified abnormality can be circled, highlighted, orotherwise indicated in their corresponding window. The medical scanassisted review system 102 can retrieve classification data for theprevious state of the abnormality by retrieving abnormality annotationdata 442 of the similar abnormality mapped to the previous scan from themedical scan database 342. This data may not be assigned to the previousscan, and the medical scan assisted review system can automaticallydetermine classification or other diagnosis data for the previousmedical scan by utilizing the medical scan image analysis system asdiscussed. Alternatively or in addition, some or all of the abnormalityclassification data 445 or other diagnosis data 440 for the previousscan can be assigned values determined based on the abnormalityclassification data or other diagnosis data determined for the currentscan. Such abnormality classification data 445 or other diagnosis data440 determined for the previous scan can be mapped to the previous scan,and or mapped to the longitudinal data 433, in the database and/ortransmitted to a responsible entity via the network.

The medical assisted review system can automatically generate statechange data such as a change in size, volume, malignancy, or otherchanges to various classifiers of the abnormality. This can be achievedby automatically comparing image data of one or more previous scans andthe current scan and/or by comparing abnormality data of the previousscan to abnormality data of the current scan. In some embodiments, suchmetrics can be calculated by utilizing the medical scan similarityanalysis function, for example, where the output of the medical scansimilarity analysis function such as the similarity score indicatesdistance, error, or other measured discrepancy in one or moreabnormality classifier categories 444 and/or abnormality patterncategories 446. This calculated distance, error, or other measureddiscrepancy in each category can be used to quantify state change data,indicate a new classifier in one or more categories, to determine if acertain category has become more or less severe, or otherwise determinehow the abnormality has changed over time. In various embodiments, thisdata can be displayed in one window, for example, where an increase inabnormality size is indicated by overlaying or highlighting an outlineof the current abnormality over the corresponding image slice of theprevious abnormality, or vice versa. In various embodiments whereseveral past scans are available, such state change data can bedetermined over time, and statistical data showing growth rate changesover time or malignancy changes over time can be generated, for example,indicating if a growth rate is lessening or worsening over time. Imageslices corresponding to multiple past scans can be displayed insequence, for example, where a first scroll bar allows a user to scrollbetween image slice numbers, and a second scroll bar allows a user toscroll between the same image slice over time. In various embodimentsthe abnormality data, heat map data, or other interface features will bedisplayed in conjunction with the image slices of the past image data.

The medical scan report labeling system 104 can be used to automaticallyassign medical codes to medical scans based on user identified keywords,phrases, or other relevant medical condition terms of natural text datain a medical scan report of the medical scan, identified by users of themedical scan report labeling system 104. The medical scan reportlabeling system 104 can be operable to transmit a medical report thatincludes natural language text to a first client device for display.Identified medical condition term data can be received from the firstclient device in response. An alias mapping pair in a medical labelalias database can be identified by determining that a medical conditionterm of the alias mapping pair compares favorably to the identifiedmedical condition term data. A medical code that corresponds to thealias mapping pair and a medical scan that corresponds to the medicalreport can be transmitted to a second client device of an expert userfor display, and accuracy data can be received from the second clientdevice in response. The medical code is mapped to the first medical scanin a medical scan database when the accuracy data indicates that themedical code compares favorably to the medical scan.

The medical scan annotator system 106 can be used to gather annotationsof medical scans based on review of the medical scan image data by usersof the system such as radiologists or other medical professionals.Medical scans that require annotation, for example, that have beentriaged from a hospital or other triaging entity, can be sent tomultiple users selected by the medical scan annotator system 106, andthe annotations received from the multiple medical professionals can beprocessed automatically by a processing system of the medical scanannotator system, allowing the medical scan annotator system toautomatically determine a consensus annotation of each medical scan.Furthermore, the users can be automatically scored by the medical scanannotator system based on how closely their annotation matches to theconsensus annotation or some other truth annotation, for example,corresponding to annotations of the medical scan assigned a truth flag.Users can be assigned automatically to annotate subsequent incomingmedical scans based on their overall scores and/or based on categorizedscores that correspond to an identified category of the incoming medicalscan.

The medical scan annotator system 106 can be operable to select amedical scan for transmission via a network to a first client device anda second client device for display via an interactive interface, andannotation data can be received from the first client device and thesecond client device in response. Annotation similarity data can begenerated by comparing the first annotation data to the secondannotation data, and consensus annotation data can be generated based onthe first annotation data and the second annotation data in response tothe annotation similarity data indicating that the difference betweenthe first annotation data and the second annotation data comparesfavorably to an annotation discrepancy threshold. The consensusannotation data can be mapped to the medical scan in a medical scandatabase.

A medical scan diagnosing system 108 can be used by hospitals, medicalprofessionals, or other medical entities to automatically produceinference data for given medical scans by utilizing computer visiontechniques and/or natural language processing techniques. Thisautomatically generated inference data can be used to generate and/orupdate diagnosis data or other corresponding data of correspondingmedical scan entries in a medical scan database. The medical scandiagnosing system can utilize a medical scan database, user database,and/or a medical scan analysis function database by communicating withthe database storage system 140 via the network 150, and/or can utilizeanother medical scan database, user database, and/or function databasestored in local memory.

The medical scan diagnosing system 108 can be operable to receive amedical scan. Diagnosis data of the medical scan can be generated byperforming a medical scan inference function on the medical scan. Thefirst medical scan can be transmitted to a first client deviceassociated with a user of the medical scan diagnosing system in responseto the diagnosis data indicating that the medical scan corresponds to anon-normal diagnosis. The medical scan can be displayed to the user viaan interactive interface displayed by a display device corresponding tothe first client device. Review data can be received from the firstclient device, where the review data is generated by the first clientdevice in response to a prompt via the interactive interface. Updateddiagnosis data can be generated based on the review data. The updateddiagnosis data can be transmitted to a second client device associatedwith a requesting entity.

A medical scan interface feature evaluating system 110 can be usedevaluate proposed interface features or currently used interfacefeatures of an interactive interface to present medical scans for reviewby medical professionals or other users of one or more subsystems 101.The medical scan interface feature evaluator system 110 can be operableto generate an ordered image-to-prompt mapping by selecting a set ofuser interface features to be displayed with each of an ordered set ofmedical scans. The set of medical scans and the ordered image-to-promptmapping can be transmitted to a set of client devices. A set ofresponses can be generated by each client device in response tosequentially displaying each of the set of medical scans in conjunctionwith a mapped user interface feature indicated in the orderedimage-to-prompt mapping via a user interface. Response score data can begenerated by comparing each response to truth annotation data of thecorresponding medical scan. Interface feature score data correspondingto each user interface feature can be generated based on aggregating theresponse score data, and is used to generate a ranking of the set ofuser interface features.

A medical scan image analysis system 112 can be used to generate and/orperform one or more medical scan image analysis functions by utilizing acomputer vision-based learning algorithm 1350 on a training set ofmedical scans with known annotation data, diagnosis data, labelingand/or medical code data, report data, patient history data, patientrisk factor data, and/or other metadata associated with medical scans.These medical scan image analysis functions can be used to generateinference data for new medical scans that are triaged or otherwiserequire inferred annotation data, diagnosis data, labeling and/ormedical code data, and/or report data. For example, some medical scanimage analysis functions can correspond to medical scan inferencefunctions of the medical scan diagnosing system or other medical scananalysis functions of a medical scan analysis function database. Themedical scan image analysis functions can be used to determine whetheror not a medical scan is normal, to detect the location of anabnormality in one or more slices of a medical scan, and/or tocharacterize a detected abnormality. The medical scan image analysissystem can be used to generate and/or perform computer vision basedmedical scan image analysis functions utilized by other subsystems ofthe medical scan processing system as described herein, aiding medicalprofessionals to diagnose patients and/or to generate further data andmodels to characterize medical scans. The medical scan image analysissystem can include a processing system that includes a processor and amemory that stores executable instructions that, when executed by theprocessing system, facilitate performance of operations.

The medical scan image analysis system 112 can be operable to receive aplurality of medical scans that represent a three-dimensional anatomicalregion and include a plurality of cross-sectional image slices. Aplurality of three-dimensional subregions corresponding to each of theplurality of medical scans can be generated by selecting a proper subsetof the plurality of cross-sectional image slices from each medical scan,and by further selecting a two-dimensional subregion from each propersubset of cross-sectional image slices. A learning algorithm can beperformed on the plurality of three-dimensional subregions to generate aneural network. Inference data corresponding to a new medical scanreceived via the network can be generated by performing an inferencealgorithm on the new medical scan by utilizing the neural network. Aninferred abnormality can be identified in the new medical scan based onthe inference data.

The medical scan natural language analysis system 114 can determine atraining set of medical scans with medical codes determined to be truthdata. Corresponding medical reports and/or other natural language textdata associated with a medical scan can be utilized to train a medicalscan natural language analysis function by generating a medical reportnatural language model. The medical scan natural language analysisfunction can be utilized to generate inference data for incoming medicalreports for other medical scans to automatically determine correspondingmedical codes, which can be mapped to corresponding medical scans.Medical codes assigned to medical scans by utilizing the medical reportnatural language model can be utilized by other subsystems, for example,to train other medical scan analysis functions, to be used as truth datato verify annotations provided via other subsystems, to aid indiagnosis, or otherwise be used by other subsystems as described herein.

A medical scan comparison system 116 can be utilized by one or moresubsystems to identify and/or display similar medical scans, forexample, to perform or determine function parameters for a medical scansimilarity analysis function, to generate or retrieve similar scan data,or otherwise compare medical scan data. The medical scan comparisonsystem 116 can also utilize some or all features of other subsystems asdescribed herein. The medical scan comparison system 116 can be operableto receive a medical scan via a network and can generate similar scandata. The similar scan data can include a subset of medical scans from amedical scan database and can be generated by performing an abnormalitysimilarity function, such as medical scan similarity analysis function,to determine that a set of abnormalities included in the subset ofmedical scans compare favorably to an abnormality identified in themedical scan. At least one cross-sectional image can be selected fromeach medical scan of the subset of medical scans for display on adisplay device associated with a user of the medical scan comparisonsystem in conjunction with the medical scan.

FIG. 2A presents an embodiment of client device 120. Each client device120 can include one or more client processing devices 230, one or moreclient memory devices 240, one or more client input devices 250, one ormore client network interfaces 260 operable to more support one or morecommunication links via the network 150 indirectly and/or directly,and/or one or more client display devices 270, connected via bus 280.Client applications 202, 204, 206, 208, 210, 212, 214, and/or 216correspond to subsystems 102, 104, 106, 108, 110, 112, 114, and/or 116of the medical scan processing system respectfully. Each client device120 can receive the application data from the corresponding subsystemvia network 150 by utilizing network interface 260, for storage in theone or more memory devices 240. In various embodiments, some or allclient devices 120 can include a computing device associated with aradiologist, medical entity, or other user of one or more subsystems asdescribed herein.

The one or more processing devices 230 can display interactive interface275 on the one or more client display devices 270 in accordance with oneor more of the client applications 202, 204, 206, 208, 210, 212, 214,and/or 216, for example, where a different interactive interface 275 isdisplayed for some or all of the client applications in accordance withthe website presented by the corresponding subsystem 102, 104, 106, 108,110, 112, 114 and/or 116. The user can provide input in response to menudata or other prompts presented by the interactive interface via the oneor more client input devices 250, which can include a microphone, mouse,keyboard, touchscreen of display device 270 itself or other touchscreen,and/or other device allowing the user to interact with the interactiveinterface. The one or more processing devices 230 can process the inputdata and/or send raw or processed input data to the correspondingsubsystem, and/or can receive and/or generate new data in response forpresentation via the interactive interface 275 accordingly, by utilizingnetwork interface 260 to communicate bidirectionally with one or moresubsystems and/or databases of the medical scan processing system vianetwork 150.

FIG. 2B presents an embodiment of a subsystem 101, which can be utilizedin conjunction with subsystem 102, 104, 106, 108, 110, 112, 114 and/or116. Each subsystem 101 can include one or more subsystem processingdevices 235, one or more subsystem memory devices 245, and/or one ormore subsystem network interfaces 265, connected via bus 285. Thesubsystem memory devices 245 can store executable instructions that,when executed by the one or more subsystem processing devices 235,facilitate performance of operations by the subsystem 101, as describedfor each subsystem herein.

FIG. 3 presents an embodiment of the database storage system 140.Database storage system 140 can include at least one database processingdevice 330, at least one database memory device 340, and at least onedatabase network interface 360, operable to more support one or morecommunication links via the network 150 indirectly and/or directly, allconnected via bus 380. The database storage system 140 can store one ormore databases the at least one memory 340, which can include a medicalscan database 342 that includes a plurality medical scan entries 352, auser database 344 that includes a plurality of user profile entries 354,a medical scan analysis function database 346 that includes a pluralityof medical scan analysis function entries 356, an interface featuredatabase 348 can include a plurality of interface feature entries 358,and/or other databases that store data generated and/or utilized by thesubsystems 101. Some or all of the databases 342, 344, 346 and/or 348can consist of multiple databases, can be stored relationally ornon-relationally, and can include different types of entries anddifferent mappings than those described herein. A database entry caninclude an entry in a relational table or entry in a non-relationalstructure. Some or all of the data attributes of an entry 352, 354, 356,and/or 358 can refer to data included in the entry itself or that isotherwise mapped to an identifier included in the entry and can beretrieved from, added to, modified, or deleted from the database storagesystem 140 based on a given identifier of the entry. Some or all of thedatabases 342, 344, 346, and/or 348 can instead be stored locally by acorresponding subsystem, for example, if they are utilized by only onesubsystem.

The processing device 330 can facilitate read/write requests receivedfrom subsystems and/or client devices via the network 150 based onread/write permissions for each database stored in the at least onememory device 340. Different subsystems can be assigned differentread/write permissions for each database based on the functions of thesubsystem, and different client devices 120 can be assigned differentread/write permissions for each database. One or more client devices 120can correspond to one or more administrators of one or more of thedatabases stored by the database storage system, and databaseadministrator devices can manage one or more assigned databases,supervise assess and/or efficiency, edit permissions, or otherwiseoversee database processes based on input to the client device viainteractive interface 275.

FIG. 4A presents an embodiment of a medical scan entry 352, stored inmedical scan database 342, included in metadata of a medical scan,and/or otherwise associated with a medical scan. A medical scan caninclude imaging data corresponding to a CT scan, x-ray, MRI, PET scan,Ultrasound, EEG, mammogram, or other type of radiological scan ormedical scan taken of an anatomical region of a human body, animal,organism, or object and further can include metadata corresponding tothe imaging data. Some or all of the medical scan entries can beformatted in accordance with a Digital Imaging and Communications inMedicine (DICOM) format or other standardized image format, and some ormore of the fields of the medical scan entry 352 can be included in aDICOM header or other standardized header of the medical scan. Medicalscans can be awaiting review or can have already been reviewed by one ormore users or automatic processes and can include tentative diagnosisdata automatically generated by a subsystem, generated based on userinput, and/or generated from another source. Some medical scans caninclude final, known diagnosis data generated by a subsystem and/orgenerated based on user input, and/or generated from another source, andcan included in training sets used to train processes used by one ormore subsystems such as the medical scan image analysis system 112and/or the medical scan natural language analysis system 114.

Some medical scans can include one or more abnormalities, which can beidentified by a user or can be identified automatically. Abnormalitiescan include nodules, for example malignant nodules identified in a chestCT scan. Abnormalities can also include and/or be characterized by oneor more abnormality pattern categories such as such as cardiomegaly,consolidation, effusion, emphysema, and/or fracture, for exampleidentified in a chest x-ray. Abnormalities can also include any otherunknown, malignant or benign feature of a medical scan identified as notnormal. Some scans can contain zero abnormalities, and can be identifiedas normal scans. Some scans identified as normal scans can includeidentified abnormalities that are classified as benign, and include zeroabnormalities classified as either unknown or malignant. Scansidentified as normal scans may include abnormalities that were notdetected by one or more subsystems and/or by an originating entity.Thus, some scans may be improperly identified as normal. Similarly,scans identified to include at least one abnormality may include atleast one abnormality that was improperly detected as an abnormality byone or more subsystems and/or by an originating entity. Thus, some scansmay be improperly identified as containing abnormalities.

Each medical scan entry 352 can be identified by its own medical scanidentifier 353, and can include or otherwise map to medical scan imagedata 410, and metadata such as scan classifier data 420, patient historydata 430, diagnosis data 440, annotation author data 450, confidencescore data 460, display parameter data 470, similar scan data 480,training set data 490, and/or other data relating to the medical scan.Some or all of the data included in a medical scan entry 352 can be usedto aid a user in generating or editing diagnosis data 440, for example,in conjunction with the medical scan assisted review system 102, themedical scan report labeling system 104, and/or the medical scanannotator system 106. Some or all of the data included in a medical scanentry 352 can be used to allow one or more subsystems 101, such asautomated portions of the medical scan report labeling system 104 and/orthe medical scan diagnosing system 108, to automatically generate and/oredit diagnosis data 440 or other data the medical scan. Some or all ofthe data included in a medical scan entry 352 can be used to train someor all medical scan analysis functions of the medical scan analysisfunction database 346 such as one or more medical scan image analysisfunctions, one or more medical scan natural language analysis functions,one or more medical scan similarity analysis functions, one or moremedical report generator functions, and/or one or more medical reportanalysis functions, for example, in conjunction with the medical scanimage analysis system 112, the medical scan natural language analysissystem 114, and/or the medical scan comparison system 116.

The medical scan entries 352 and the associated data as described hereincan also refer to data associated with a medical scan that is not storedby the medical scan database, for example, that is uploaded by a clientdevice for direct transmission to a subsystem, data generated by asubsystem and used as input to another subsystem or transmitted directlyto a client device, data stored by a Picture Archive and CommunicationSystem (PACS) communicating with the medical scan processing system 100,or other data associated with a medical scan that is received and orgenerated without being stored in the medical scan database 342. Forexample, some or all of the structure and data attributes described withrespect to a medical scan entry 352 can also correspond to structureand/or data attribute of data objects or other data generated by and/ortransmitted between subsystems and/or client devices that correspond toa medical scan. Herein, any of the data attributes described withrespect to a medical scan entry 352 can also correspond to dataextracted from a data object generated by a subsystem or client deviceor data otherwise received from a subsystem, client device, or othersource via network 150 that corresponds to a medical scan.

The medical scan image data 410 can include one or more imagescorresponding to a medical scan. The medical scan image data 410 caninclude one or more image slices 412, for example, corresponding to asingle x-ray image, a plurality of cross-sectional, tomographic imagesof a scan such as a CT scan, or any plurality of images taken from thesame or different point at the same or different angles. The medicalscan image data 410 can also indicate an ordering of the one or moreimage slices 412. Herein, a “medical scan” can refer a full scan of anytype represented by medical scan image data 410. Herein, an “imageslice” can refer to one of a plurality of cross-sectional images of themedical scan image data 410, one of a plurality of images taken fromdifferent angles of the medical scan image data 410, and/or the singleimage of the medical scan image data 410 that includes only one image.Furthermore “plurality of image slices” can refer to all of the imagesof the associated medical scan, and refers to only a single image if themedical scan image data 410 includes only one image. Each image slice412 can include a plurality of pixel values 414 mapped to each pixel ofthe image slice. Each pixel value can correspond to a density value,such as a Hounsfield value or other measure of density. Pixel values canalso correspond to a grayscale value, a RGB (Red-Green-Blue) or othercolor value, or other data stored by each pixel of an image slice 412.

Scan classifier data 420 can indicate classifying data of the medicalscan. Scan classifier data can include scan type data 421, for example,indicating the modality of the scan. The scan classifier data canindicate that the scan is a CT scan, x-ray, MRI, PET scan, Ultrasound,EEG, mammogram, or other type of scan. Scan classifier data 420 can alsoinclude anatomical region data 422, indicating for example, the scan isa scan of the chest, head, right knee, or other anatomical region. Scanclassifier data can also include originating entity data 423, indicatingthe hospital where the scan was taken and/or a user that uploaded thescan to the system. If the originating entity data corresponds to a userof one or more subsystems 101, the originating entity data can include acorresponding user profile identifier and/or include other data from theuser profile entry 354 of the user. Scan classifier data 420 can includegeographic region data 424, indicating a city, state, and/or countryfrom which the scan originated, for example, based on the user dataretrieved from the user database 344 based on the originating entity.Scan classifier data can also include machine data 425, which caninclude machine identifier data, machine model data, machine calibrationdata, and/or contrast agent data, for example based on imaging machinedata retrieved from the user database 344 based on the originatingentity data 423. The scan classifier data 420 can include scan date data426 indicating when the scan was taken. The scan classifier data 420 caninclude scan priority data 427, which can indicate a priority score,ranking, number in a queue, or other priority data with regard totriaging and/or review. A priority score, ranking, or queue number ofthe scan priority data 427 can be generated by automatically by asubsystem based on the scan priority data 427, based on a severity ofpatient symptoms or other indicators in the risk factor data 432, basedon a priority corresponding to the originating entity, based onpreviously generated diagnosis data 440 for the scan, and/or can beassigned by the originating entity and/or a user of the system.

The scan classifier data 420 can include other classifying data notpictured in FIG. 4A. For example, a set of scans can include medicalscan image data 410 corresponding to different imaging planes. The scanclassifier data can further include imaging plane data indicating one ormore imaging planes corresponding to the image data. For example, theimaging plane data can indicate the scan corresponds to the axial plane,sagittal plane, or coronal plane. A single medical scan entry 352 caninclude medical scan image data 410 corresponding multiple planes, andeach of these planes can be tagged appropriately in the image data. Inother embodiments, medical scan image data 410 corresponding to eachplane can be stored as separate medical scan entries 352, for example,with a common identifier indicating these entries belong to the same setof scans.

Alternatively or in addition, the scan classifier data 420 can includesequencing data. For example, a set of scans can include medical scanimage data 410 corresponding to different sequences. The scan classifierdata can further include sequencing data indicating one or more of aplurality of sequences of the image data corresponds to, for example,indicating whether an MRI scan corresponds to a T2 sequence, a T1sequence, a T1 sequence with contrast, a diffusion sequence, a FLAIRsequence, or other MRI sequence. A single medical scan entry 352 caninclude medical scan image data 410 corresponding to multiple sequences,and each of these sequences can be tagged appropriately in the entry. Inother embodiments, medical scan image data 410 corresponding to eachsequence can be stored as separate medical scan entries 352, forexample, with a common identifier indicating these entries belong to thesame set of scans.

Alternatively or in addition, the scan classifier data 420 can includean image quality score. This score can be determined automatically byone or more subsystems 101, and/or can be manually assigned the medicalscan. The image quality score can be based on a resolution of the imagedata 410, where higher resolution image data is assigned a morefavorable image quality score than lower resolution image data. Theimage quality score can be based on whether the image data 410corresponds to digitized image data received directly from thecorresponding imaging machine, or corresponds to a hard copy of theimage data that was later scanned in. In some embodiments, the imagequality score can be based on a detected corruption, and/or detectedexternal factor that determined to negatively affect the quality of theimage data during the capturing of the medical scan and/or subsequent tothe capturing of the medical scan. In some embodiments, the imagequality score can be based on detected noise in the image data, where amedical scan with a higher level of detected noise can receive a lessfavorable image quality score than a medical scan with a lower level ofdetected noise. Medical scans with this determined corruption orexternal factor can receive a less favorable image quality score thanmedical scans with no detected corruption or external factor.

In some embodiments, the image quality score can be based on includemachine data 425. In some embodiments, one or more subsystems canutilize the image quality score to flag medical scans with image qualityscores that fall below an image quality threshold. The image qualitythreshold can be the same or different for different subsystems, medicalscan modalities, and/or anatomical regions. For example, the medicalscan image analysis system can automatically filter training sets basedon selecting only medical scans with image quality scores that comparefavorably to the image quality threshold. As another example, one ormore subsystems can flag a particular imaging machine and/or hospital orother medical entity that have produced at least a threshold numberand/or percentage of medical scan with image quality scores that compareunfavorably to the image quality threshold. As another example, ade-noising algorithm can be automatically utilized to clean the imagedata when the image quality score compares unfavorably to the imagequality threshold. As another example, the medical scan image analysissystem can select a particular medical image analysis function from aset of medical image analysis functions to utilize on a medical scan togenerate inference data for the medical scan. Each of this set ofmedical image analysis function can be trained on different levels ofimage quality, and the selected image analysis function can be selectedbased on the determined image quality score falling within a range ofimage quality scores the image analysis function was trained on and/oris otherwise suitable for.

The patient history data 430 can include patient identifier data 431which can include basic patient information such as name or anidentifier that may be anonymized to protect the confidentiality of thepatient, age, and/or gender. The patient identifier data 431 can alsomap to a patient entry in a separate patient database stored by thedatabase storage system, or stored elsewhere. The patient history datacan include patient risk factor data 432 which can include previousmedical history, family medical history, smoking and/or drug habits,pack years corresponding to tobacco use, environmental exposures,patient symptoms, etc. The patient history data 430 can also includelongitudinal data 433, which can identify one or more additional medicalscans corresponding to the patient, for example, retrieved based onpatient identifier data 431 or otherwise mapped to the patientidentifier data 431. Some or all additional medical scans can beincluded in the medical scan database, and can be identified based ontheir corresponding identifiers medical scan identifiers 353. Some orall additional medical scans can be received from a different source andcan otherwise be identified. Alternatively or in addition, thelongitudinal data can simply include some or all relevant scan entrydata of a medical scan entry 352 corresponding to the one or moreadditional medical scans. The additional medical scans can be the sametype of scan or different types of scans. Some or all of the additionalscans may correspond to past medical scans, and/or some or all of theadditional scans may correspond to future medical scans. Thelongitudinal data 433 can also include data received and/or determinedat a date after the scan such as final biopsy data, or some or all ofthe diagnosis data 440. The patient history data can also include alongitudinal quality score 434, which can be calculated automatically bya subsystem, for example, based on the number of additional medicalscans, based on how many of the additional scans in the file were takenbefore and/or after the scan based on the scan date data 426 of themedical scan and the additional medical scans, based on a date rangecorresponding to the earliest scan and corresponding to the latest scan,based on the scan types data 421 these scans, and/or based on whether ornot a biopsy or other final data is included. As used herein, a “high”longitudinal quality score refers to a scan having more favorablelongitudinal data than that with a “low” longitudinal quality score.

Diagnosis data 440 can include data that indicates an automateddiagnosis, a tentative diagnosis, and/or data that can otherwise be usedto support medical diagnosis, triage, medical evaluation and/or otherreview by a medical professional or other user. The diagnosis data 440of a medical scan can include a binary abnormality identifier 441indicating whether the scan is normal or includes at least oneabnormality. In some embodiments, the binary abnormality identifier 441can be determined by comparing some or all of confidence score data 460to a threshold, can be determined by comparing a probability value to athreshold, and/or can be determined by comparing another continuous ordiscrete value indicating a calculated likelihood that the scan containsone or more abnormalities to a threshold. In some embodiments,non-binary values, such as one or more continuous or discrete valuesindicating a likelihood that the scan contains one or moreabnormalities, can be included in diagnosis data 440 in addition to, orinstead of, binary abnormality identifier 441. One or abnormalities canbe identified by the diagnosis data 440, and each identified abnormalitycan include its own set of abnormality annotation data 442.Alternatively, some or all of the diagnosis data 440 can indicate and/ordescribe multiple abnormalities, and thus will not be presented for eachabnormality in the abnormality annotation data 442. For example, thereport data 449 of the diagnosis data 440 can describe all identifiedabnormalities, and thus a single report can be included in thediagnosis.

FIG. 4B presents an embodiment of the abnormality annotation data 442.The abnormality annotation data 442 for each abnormality can includeabnormality location data 443, which can include an anatomical locationand/or a location specific to pixels, image slices, coordinates or otherlocation information identifying regions of the medical scan itself. Theabnormality annotation data 442 can include abnormality classificationdata 445 which can include binary, quantitative, and/or descriptive dataof the abnormality as a whole, or can correspond to one or moreabnormality classifier categories 444, which can include size, volume,pre-post contrast, doubling time, calcification, components, smoothness,spiculation, lobulation, sphericity, internal structure, texture, orother categories that can classify and/or otherwise characterize anabnormality. Abnormality classifier categories 444 can be assigned abinary value, indicating whether or not such a category is present. Forexample, this binary value can be determined by comparing some or all ofconfidence score data 460 to a threshold, can be determined by comparinga probability value to a threshold, and/or can be determined bycomparing another continuous or discrete value indicating a calculatedlikelihood that a corresponding abnormality classifier category 444 ispresent to a threshold, which can be the same or different threshold foreach abnormality classifier category 444. In some embodiments,abnormality classifier categories 444 can be assigned one or morenon-binary values, such as one or more continuous or discrete valuesindicating a likelihood that the corresponding classifier category 444is present.

The abnormality classifier categories 444 can also include a malignancycategory, and the abnormality classification data 445 can include amalignancy rating such as a Lung-RADS score, a Fleischner score, and/orone or more calculated values that indicate malignancy level, malignancyseverity, and/or probability of malignancy. Alternatively or inaddition, the malignancy category can be assigned a value of “yes”,“no”, or “maybe”. The abnormality classifier categories 444 can alsoinclude abnormality pattern categories 446 such as cardiomegaly,consolidation, effusion, emphysema, and/or fracture, and the abnormalityclassification data 445 for each abnormality pattern category 446 canindicate whether or not each of the abnormality patterns is present.

The abnormality classifier categories can correspond to ResponseEvaluation Criteria in Solid Tumors (RECIST) eligibility and/or RECISTevaluation categories. For example, an abnormality classifier category444 corresponding to RECIST eligibility can have correspondingabnormality classification data 445 indicating a binary value “yes” or“no”, and/or can indicate if the abnormality is a “target lesion” and/ora “non-target lesion.” As another example, an abnormality classifiercategory 444 corresponding to a RECIST evaluation category can bedetermined based on longitudinal data 433 and can have correspondingabnormality classification data 445 that includes one of the set ofpossible values “Complete Response”, “Partial Response”, “StableDisease”, or “Progressive Disease.”

The diagnosis data 440 as a whole, and/or the abnormality annotationdata 442 for each abnormality, can include custom codes or datatypesidentifying the binary abnormality identifier 441, abnormality locationdata 443 and/or some or all of the abnormality classification data 445of one or more abnormality classifier categories 444. Alternatively orin addition, some or all of the abnormality annotation data 442 for eachabnormality and/or other diagnosis data 440 can be presented in a DICOMformat or other standardized image annotation format, and/or can beextracted into custom datatypes based on abnormality annotation dataoriginally presented in DICOM format. Alternatively or in addition, thediagnosis data 440 and/or the abnormality annotation data 442 for eachabnormality can be presented as one or more medical codes 447 such asSNOMED codes, Current Procedure Technology (CPT) codes, ICD-9 codes,ICD-10 codes, or other standardized medical codes used to label orotherwise describe medical scans.

Alternatively or in addition, the diagnosis data 440 can include naturallanguage text data 448 annotating or otherwise describing the medicalscan as a whole, and/or the abnormality annotation data 442 can includenatural language text data 448 annotating or otherwise describing eachcorresponding abnormality. In some embodiments, some or all of thediagnosis data 440 is presented only as natural language text data 448.In some embodiments, some or all of the diagnosis data 440 isautomatically generated by one or more subsystems based on the naturallanguage text data 448, for example, without utilizing the medical scanimage data 410, for example, by utilizing one or more medical scannatural language analysis functions trained by the medical scan naturallanguage analysis system 114. Alternatively or in addition, someembodiments, some or all of the natural language text data 448 isgenerated automatically based on other diagnosis data 440 such asabnormality annotation data 442, for example, by utilizing a medicalscan natural language generating function trained by the medical scannatural language analysis system 114.

The diagnosis data can include report data 449 that includes at leastone medical report, which can be formatted to include some or all of themedical codes 447, some or all of the natural language text data 448,other diagnosis data 440, full or cropped images slices formatted basedon the display parameter data 470 and/or links thereto, full or croppedimages slices or other data based on similar scans of the similar scandata 480 and/or links thereto, full or cropped images or other databased on patient history data 430 such as longitudinal data 433 and/orlinks thereto, and/or other data or links to data describing the medicalscan and associated abnormalities. The diagnosis data 440 can alsoinclude finalized diagnosis data corresponding to future scans and/orfuture diagnosis for the patient, for example, biopsy data or otherlongitudinal data 433 determined subsequently after the scan. Themedical report of report data 449 can be formatted based on specifiedformatting parameters such as font, text size, header data, bulleting ornumbering type, margins, file type, preferences for including one ormore full or cropped image slices 412, preferences for including similarmedical scans, preferences for including additional medical scans, orother formatting to list natural language text data and/or image data,for example, based on preferences of a user indicated in the originatingentity data 423 or other responsible user in the corresponding reportformatting data.

Annotation author data 450 can be mapped to the diagnosis data for eachabnormality, and/or mapped to the scan as a whole. This can include oneor more annotation author identifiers 451, which can include one or moreuser profile identifiers of a user of the system, such as an individualmedical professional, medical facility and/or medical entity that usesthe system. Annotation author data 450 can be used to determine theusage data of a user profile entry 354. Annotation author data 450 canalso include one or more medical scan analysis function identifiers 357or other function identifier indicating one or more functions or otherprocesses of a subsystem responsible for automatically generating and/orassisting a user in generating some or all of the diagnosis data, forexample an identifier of a particular type and/or version of a medicalscan image analysis functions that was used by the medical scandiagnosing system 108 used to generate part or all of the diagnosis data440 and/or an interface feature identifier, indicating an one or moreinterface features presented to a user to facilitate entry of and/orreviewing of the diagnosis data 440. The annotation author data can alsosimply indicate, for one or more portions of the diagnosis data 440, ifthis portion was generated by a human or automatically generated by asubsystem of the medical scan processing system.

In some embodiments, if a medical scan was reviewed by multipleentities, multiple, separate diagnosis data entries 440 can be includedin the medical scan entry 352, mapped to each diagnosis author in theannotation author data 450. This allows different versions of diagnosisdata 440 received from multiple entities. For example, annotation authordata of a particular medical scan could indicate that the annotationdata was written by a doctor at medical entity A, and the medical codedata was generated by user Y by utilizing the medical scan reportlabeling system 104, which was confirmed by expert user X. Theannotation author data of another medical scan could indicate that themedical code was generated automatically by utilizing version 7 of themedical scan image analysis function relating to chest x-rays, andconfirmed by expert user X. The annotation author data of anothermedical scan could indicate that the location and a first malignancyrating were generated automatically by utilizing version 7 of themedical scan image analysis function relating to chest x-rays, and thata second malignancy rating was entered by user Z. In some embodiments,one of the multiple diagnosis entries can include consensus annotationdata, for example, generated automatically by a subsystem such as themedical scan annotating system 106 based on the multiple diagnosis data440, based on confidence score data 460 of each of the multiplediagnosis data 440, and/or based on performance score data of acorresponding user, a medical scan analysis function, or an interfacefeature, identified in the annotation author data for each correspondingone of the multiple diagnosis data 440.

Confidence score data 460 can be mapped to some or all of the diagnosisdata 440 for each abnormality, and/or for the scan as a whole. This caninclude an overall confidence score for the diagnosis, a confidencescore for the binary indicator of whether or not the scan was normal, aconfidence score for the location a detected abnormality, and/orconfidence scores for some or all of the abnormality classifier data.This may be generated automatically by a subsystem, for example, basedon the annotation author data and corresponding performance score of oneor more identified users and/or subsystem attributes such as interactiveinterface types or medical scan image analysis functions indicated bythe annotation author data. In the case where multiple diagnosis dataentries 440 are included from different sources, confidence score data460 can be computed for each entry and/or an overall confidence score,for example, corresponding to consensus diagnosis data, can be based oncalculated distance or other error and/or discrepancies between theentries, and/or can be weighted on the confidence score data 460 of eachentry. In various embodiments, the confidence score data 460 can includea truth flag 461 indicating the diagnosis data is considered as “known”or “truth”, for example, flagged based on user input, flaggedautomatically based on the author data, and/or flagged automaticallybased on the calculated confidence score of the confidence score dataexceeding a truth threshold. As used herein, a “high” confidence scorerefers to a greater degree or more favorable level of confidence than a“low” confidence score.

Display parameter data 470 can indicate parameters indicating an optimalor preferred display of the medical scan by an interactive interface 275and/or formatted report for each abnormality and/or for the scan as awhole. Some or all of the display parameter data can have separateentries for each abnormality, for example, generated automatically by asubsystem 101 based on the abnormality annotation data 442. Displayparameter data 470 can include interactive interface feature data 471,which can indicate one or more selected interface features associatedwith the display of abnormalities and/or display of the medical scan asa whole, and/or selected interface features associated with userinteraction with a medical scan, for example, based on categorizedinterface feature performance score data and a category associated withthe abnormality and/or with the medical scan itself. The displayparameter data can include a slice subset 472, which can indicate aselected subset of the plurality of image slices that includes a singleimage slice 412 or multiple image slices 412 of the medical scan imagedata 410 for display by a user interface. The display parameter data 470can include slice order data 473 that indicates a selected customordering and/or ranking for the slice subset 472, or for all of theslices 412 of the medical scan. The display parameter data 470 caninclude slice cropping data 474 corresponding to some or all of theslice subset 472, or all of the image slices 412 of the medical scan,and can indicating a selected custom cropped region of each image slice412 for display, or the same selected custom cropped region for theslice subset 472 or for all slices 412. The display parameter data caninclude density window data 475, which can indicate a selected customdensity window for display of the medical scan as a whole, a selectedcustom density window for the slices subset 472, and/or selected customdensity windows for each of the image slices 412 of the slice subset472, and/or for each image slice 412 of the medical scan. The densitywindow data 475 can indicate a selected upper density value cut off anda selected lower density value cut off, and/or can include a selecteddeterministic function to map each density value of a pixel to agrayscale value based on the preferred density window. The interactiveinterface feature data 471, slice subset 472, slice order data 473,slice cropping data 474, and/or the density window data 475 can beselected via user input and/or generated automatically by one or moresubsystems 101, for example, based on the abnormality annotation data442 and/or based on performance score data of different interactiveinterface versions.

Similar scan data 480 can be mapped to each abnormality, or the scan asa whole, and can include similar scan identifier data 481 correspondingto one or more identified similar medical scans, for example,automatically identified by a subsystem 101, for example, by applying asimilar scan identification step of the medical scan image analysissystem 112 and/or applying medical scan similarity analysis function tosome or all of the data stored in the medical scan entry of the medicalscan, and/or to some or all corresponding data of other medical scans inthe medical scan database. The similar scan data 480 can also correspondto medical scans received from another source. The stored similaritydata can be used to present similar cases to users of the system and/orcan be used to train medical scan image analysis functions or medicalscan similarity analysis functions.

Each identified similar medical scan can have its own medical scan entry352 in the medical scan database 342 with its own data, and the similarscan identifier data 481 can include the medical scan identifier 353each similar medical scan. Each identified similar medical scan can be ascan of the same scan type or different scan type than medical scan.

The similar scan data 480 can include a similarity score 482 for eachidentified similar scan, for example, generated based on some or all ofthe data of the medical scan entry 352 for medical scan and based onsome or all of the corresponding data of the medical scan entry 352 forthe identified similar medical scan. For example, the similarity score482 can be generated based on applying a medical scan similarityanalysis function to the medical image scan data of medical scans and402, to some or all of the abnormality annotation data of medical scansand 402, and/or to some or all of the patient history data 430 ofmedical scans and 402 such as risk factor data 432. As used herein, a“high” similarity score refers a higher level of similarity that a “low”similarity score.

The similar scan data 480 can include its own similar scan displayparameter data 483, which can be determined based on some or all of thedisplay parameter data 470 of the identified similar medical scan. Someor all of the similar scan display parameter data 483 can be generatedautomatically by a subsystem, for example, based on the displayparameter data 470 of the identified similar medical scan, based on theabnormality annotation data 442 of the medical scan itself and/or basedon display parameter data 470 of the medical scan itself. Thus, thesimilar scan display parameter data 483 can be the same or differentthan the display parameter data 470 mapped to the identified similarmedical scan and/or can be the same or different than the displayparameter data 470 of the medical scan itself. This can be utilized whendisplaying similar scans to a user via interactive interface 275 and/orcan be utilized when generating report data 449 that includes similarscans, for example, in conjunction with the medical scan assisted reviewsystem 102.

The similar scan data 480 can include similar scan abnormality data 484,which can indicate one of a plurality of abnormalities of the identifiedsimilar medical scan and its corresponding abnormality annotation data442. For example, the similarity scan abnormality data 484 can includean abnormality pair that indicates one of a plurality of abnormalitiesof the medical scan, and indicates one of a plurality of abnormalitiesof the identified similar medical scan, for example, that was identifiedas the similar abnormality.

The similar scan data 480 can include similar scan filter data 485. Thesimilar scan filter data can be generated automatically by a subsystem,and can include a selected ordered or un-ordered subset of allidentified similar scans of the similar scan data 480, and/or a rankingof all identified similar scans. For example, the subset can be selectedand/or some or all identified similar scans can be ranked based on eachsimilarity score 482, and/or based on other factors such as based on alongitudinal quality score 434 of each identified similar medical scan.

The training set data 490 can indicate one or more training sets thatthe medical scan belongs to. For example, the training set data canindicate one or more training set identifiers 491 indicating one or moremedical scan analysis functions that utilized the medical scan in theirtraining set, and/or indicating a particular version identifier 641 ofthe one or more medical scan analysis functions that utilized themedical scan in their training set. The training set data 490 can alsoindicate which portions of the medical scan entry were utilized by thetraining set, for example, based on model parameter data 623 of thecorresponding medical scan analysis functions. For example, the trainingset data 490 can indicate that the medical scan image data 410 wasincluded in the training set utilized to train version X of the chestx-ray medical scan image analysis function, or that the natural languagetext data 448 of this medical scan was used to train version Y of thenatural language analysis function.

FIG. 5A presents an embodiment of a user profile entry 354, stored inuser database 344 or otherwise associated with a user. A user cancorrespond to a user of one or more of the subsystems such as aradiologist, doctor, medical professional, medical report labeler,administrator of one or more subsystems or databases, or other user thatuses one or more subsystems 101. A user can also correspond to a medicalentity such as a hospital, medical clinic, establishment that utilizesmedical scans, establishment that employs one or more of the medicalprofessionals described, an establishment associated with administeringone or more subsystems, or other entity. A user can also correspond to aparticular client device 120 or account that can be accessed one or moremedical professionals or other employees at the same or differentmedical entities. Each user profile entry can have a corresponding userprofile identifier 355.

A user profile entry 354 can include basic user data 510, which caninclude identifying information 511 corresponding to the user such as aname, contact information, account/login/password information,geographic location information such as geographic region data 424,and/or other basic information. Basic user data 510 can includeaffiliation data 512, which can list one or more medical entities orother establishments the user is affiliated with, for example, if theuser corresponds to a single person such as a medical professional, orif the user corresponds to a hospital in a network of hospitals. Theaffiliation data 512 can include one or more corresponding user profileidentifiers 355 and/or basic user data 510 if the correspondingaffiliated medical entity or other establishment has its own entry inthe user database. The user identifier data can include employee data513 listing one or more employees, such as medical professionals withtheir own user profile entries 354, for example, if the user correspondsto a medical entity or supervising medical professional of other medicalprofessional employees, and can list a user profile identifier 355and/or basic user data 510 for each employee. The basic user data 510can also include imaging machine data 514, which can include a list ofmachines affiliated with the user which can include machine identifiers,model information, calibration information, scan type information, orother data corresponding to each machine, for example, corresponding tothe machine data 425. The user profile entry can include client devicedata 515, which can include identifiers for one or more client devicesassociated with the user, for example, allowing subsystems 101 to senddata to a client device 120 corresponding to a selected user based onthe client device data and/or to determine a user that data was receivedby determining the client device from which the data was received.

The user profile entry can include usage data 520 which can includeidentifying information for a plurality of usages by the user inconjunction with using one or more subsystems 101. This can includeconsumption usage data 521, which can include a listing of, or aggregatedata associated with, usages of one or more subsystems by the user, forexample, where the user is utilizing the subsystem as a service. Forexample, the consumption usage data 521 can correspond to each instancewhere diagnosis data was sent to the user for medical scans provided tothe user in conjunction with the medical scan diagnosing system 108and/or the medical scan assisted review system 102. Some or all ofconsumption usage data 521 can include training usage data 522,corresponding to usage in conjunction with a certification program orother user training provided by one or more subsystems. The trainingusage data 522 can correspond to each instance where diagnosis feedbackdata was provided by user for a medical scan with known diagnosis data,but diagnosis feedback data is not utilized by a subsystem to generate,edit, and/or confirm diagnosis data 440 of the medical scan, as it isinstead utilized to train a user and/or determine performance data for auser.

Usage data 520 can include contribution usage data 523, which caninclude a listing of, or aggregate data associated with, usages of oneor more subsystems 101 by the user, for example, where the user isgenerating and/or otherwise providing data and/or feedback that can isutilized by the subsystems, for example, to generate, edit, and/orconfirm diagnosis data 440 and/or to otherwise populate, modify, orconfirm portions of the medical scan database or other subsystem data.For example, the contribution usage data 523 can correspond to diagnosisfeedback data received from user, used to generate, edit, and/or confirmdiagnosis data. The contribution usage data 523 can include interactiveinterface feature data 524 corresponding to the interactive interfacefeatures utilized with respect to the contribution.

The consumption usage data 521 and/or the contribution usage data 523can include medical scan entry 352 whose entries the user utilizedand/or contributed to, can indicate one or more specific attributes of amedical scan entry 352 that a user utilized and/or contributed to,and/or a log of the user input generated by a client device of the userin conjunction with the data usage. The contribution usage data 523 caninclude the diagnosis data that the user may have generated and/orreviewed, for example, indicated by, mapped to, and/or used to generatethe annotation author data 450 of corresponding medical scan entries352. Some usages may correspond to both consumption usage of theconsumption usage data 521 and contribution usage of the contributionusage data 523. The usage data 520 can also indicate one or moresubsystems 101 that correspond to each consumption and/or contribution.

The user profile entry can include performance score data 530. This caninclude one or more performance scores generated based on thecontribution usage data 523 and/or training usage data 522. Theperformance scores can include separate performance scores generated forevery contribution in the contribution usage data 523 and/or trainingusage data 522 and/or generated for every training consumption usagescorresponding to a training program. As used herein, a “high”performance score refers to a more favorable performance or rating thana “low” performance score.

The performance score data can include accuracy score data 531, whichcan be generated automatically by a subsystem for each contribution, forexample, based on comparing diagnosis data received from a user to datato known truth data such as medical scans with a truth flag 461, forexample, retrieved from the corresponding medical scan entry 352 and/orbased on other data corresponding to the medical scan, for example,received from an expert user that later reviewed the contribution usagedata of the user and/or generated automatically by a subsystem. Theaccuracy score data 531 can include an aggregate accuracy scoregenerated automatically by a subsystem, for example, based on theaccuracy data of multiple contributions by the user over time.

The performance data can also include efficiency score data 532generated automatically by a subsystem for each contribution based on anamount of time taken to complete a contribution, for example, from atime the request for a contribution was sent to the client device to atime that the contribution was received from the client device, based ontiming data received from the client device itself, and/or based onother factors. The efficiency score can include an aggregate efficiencyscore, which can be generated automatically by a subsystem based on theindividual efficiency scores over time and/or based on determining acontribution completion rate, for example based on determining how manycontributions were completed in a fixed time window.

Aggregate performance score data 533 can be generated automatically by asubsystem based on the aggregate efficiency and/or accuracy data. Theaggregate performance data can include categorized performance data 534,for example, corresponding to different scan types, different anatomicalregions, different subsystems, different interactive interface featuresand/or display parameters. The categorized performance data 534 can bedetermined automatically by a subsystem based on the scan type data 421and/or anatomical region data 422 of the medical scan associated witheach contribution, one or more subsystems 101 associated with eachcontribution, and/or interactive interface feature data 524 associatedwith each contribution. The aggregate performance data can also be basedon performance score data 530 of individual employees if the usercorresponds to a medical entity, for example, retrieved based on userprofile identifiers 355 included in the employee data 513. Theperformance score data can also include ranking data 535, which caninclude an overall ranking or categorized rankings, for example,generated automatically by a subsystem or the database itself based onthe aggregate performance data.

In some embodiments, aggregate data for each user can be further brokendown based on scores for distinct scan categories, for example, based onthe scan classifier data 420, for example, where a first aggregate datascore is generated for a user “A” based on scores from all knee x-rays,and a second aggregate data score is generated for user A based onscores from all chest CT scans. Aggregate data for each user can befurther based on scores for distinct diagnosis categories, where a firstaggregate data score is generated for user A based on scores from allnormal scans, and a second aggregate data score is generated for user Abased on scores from all scans that contain an abnormality. This can befurther broken down, where a first aggregate score is generated for userA based on all scores from scans that contain an abnormality of a firsttype and/or in a first anatomical location, and a second aggregate scoreis generated for A based on all scores from scans that contain anabnormality of a second type and/or in a second location. Aggregate datafor each user can be further based on affiliation data, where a rankingis generated for a medical professional “B” based on scores from allmedical professionals with the same affiliation data, and/or where aranking is generated for a hospital “C” based on scores for allhospitals, all hospitals in the same geographical region, etc. Aggregatedata for each user can be further based on scores for interfacefeatures, where a first aggregate data score is generated for user Abased on scores using a first interface feature, and a second aggregatedata score is generated for user A based on scores using a firstinterface feature.

The user profile entry can include qualification data 540. Thequalification data can include experience data 541 such as educationdata, professional practice data, number of years practicing, awardsreceived, etc. The qualification data 540 can also include certificationdata 542 corresponding to certifications earned based on contributionsto one or more subsystems, for example, assigned to users automaticallyby a subsystem based on the performance score data 530 and/or based on anumber of contributions in the contribution usage data 523 and/ortraining usage data 522. For example, the certifications can correspondto standard and/or recognized certifications to train medicalprofessionals and/or incentivize medical professionals to use thesystem. The qualification data 540 can include expert data 543. Theexpert data 543 can include a binary expert identifier, which can begenerated automatically by a subsystem based on experience data 541,certification data 542, and/or the performance score data 530, and canindicate whether the user is an expert user. The expert data 543 caninclude a plurality of categorized binary expert identifierscorresponding to a plurality of qualification categories correspondingto corresponding to scan types, anatomical regions, and/or theparticular subsystems. The categorized binary expert identifiers can begenerated automatically by a subsystem based on the categorizedperformance data 534 and/or the experience data 541. The categories beranked by performance score in each category to indicate particularspecialties. The expert data 543 can also include an expert ranking orcategorized expert ranking with respect to all experts in the system.

The user profile entry can include subscription data 550, which caninclude a selected one of a plurality of subscription options that theuser has subscribed to. For example, the subscription options cancorrespond to allowed usage of one or more subsystems, such as a numberof times a user can utilize a subsystem in a month, and/or to acertification program, for example paid for by a user to receivetraining to earn a subsystem certification of certification data 542.The subscription data can include subscription expiration information,and/or billing information. The subscription data can also includesubscription status data 551, which can for example indicate a number ofremaining usages of a system and/or available credit information. Forexample, the remaining number of usages can decrease and/or availablecredit can decrease in response to usages that utilize one or moresubsystems as a service, for example, indicated in the consumption usagedata 521 and/or training usage data 522. In some embodiments, theremaining number of usages can increase and/or available credit canincrease in response to usages that correspond to contributions, forexample, based on the contribution usage data 523. An increase in creditcan be variable, and can be based on a determined quality of eachcontribution, for example, based on the performance score data 530corresponding to the contribution where a higher performance scorecorresponds to a higher increase in credit, based on scan priority data427 of the medical scan where contributing to higher priority scanscorresponds to a higher increase in credit, or based on other factors.

The user profile entry 354 can include interface preference data 560.The interface preference data can include a preferred interactiveinterface feature set 561, which can include one or more interactiveinterface feature identifiers and/or one or more interactive interfaceversion identifiers of interface feature entries 358 and/or versionidentifiers of the interface features. Some or all of the interfacefeatures of the preferred interactive interface feature set 561 cancorrespond to display parameter data 470 of medical scans. The preferredinteractive interface feature set 561 can include a single interactivefeature identifier for one or more feature types and/or interface types,and/or can include a single interactive interface version identifier forone or more interface categories. The preferred interactive interfacefeature set 561 can include a ranking of multiple features for the samefeature type and/or interface type. The ranked and/or unranked preferredinteractive interface feature set 561 can be generated based on userinput to an interactive interface of the client device to select and/orrank some or all of the interface features and/or versions. Some or allof the features and/or versions of the preferred interactive feature setcan be selected and/or ranked automatically by a subsystem such as themedical scan interface evaluator system, for example based on interfacefeature performance score data and/or feature popularity data.Alternatively or in addition, the performance score data 530 can beutilized by a subsystem to automatically determine the preferredinteractive feature set, for example, based on the scores in differentfeature-based categories of the categorized performance data 534.

The user profile entry 354 can include report formatting data 570, whichcan indicate report formatting preferences indicated by the user. Thiscan include font, text size, header data, bulleting or numbering type,margins, file type, preferences for including one or more full orcropped image slices 412, preferences for including similar medicalscans, preferences for including additional medical scans in reports, orother formatting preference to list natural language text data and/orimage data corresponding to each abnormality. Some or all of the reportformatting data 570 can be based on interface preference data 560. Thereport formatting data 570 can be used by one or more subsystems toautomatically generate report data 449 of medical scans based on thepreferences of the requesting user.

FIG. 5B presents an embodiment of a medical scan analysis function entry356, stored in medical scan analysis function database 346 or otherwiseassociated with one of a plurality of medical scan analysis functionstrained by and/or utilized by one or more subsystems 101. For example, amedical scan analysis function can include one or more medical scanimage analysis functions trained by the medical scan image analysissystem 112; one or more medical scan natural language analysis functionstrained by the medical scan natural language analysis system 114; one ormore medical scan similarity analysis function trained by the medicalscan image analysis system 112, the medical scan natural languageanalysis system 114, and/or the medical scan comparison system 116; oneor more medical report generator functions trained by the medical scannatural language analysis system 114 and/or the medical scan imageanalysis system 112, and/or the medical report analysis function trainedby the medical scan natural language analysis system 114. Some or all ofthe medical scan analysis functions can correspond to medical scaninference functions of the medical scan diagnosing system 108, thede-identification function and/or the inference functions utilized by amedical picture archive integration system as discussed in conjunctionwith FIGS. 8A-8F, or other functions and/or processes described hereinin conjunction with one or more subsystems 101. Each medical scananalysis function entry 356 can include a medical scan analysis functionidentifier 357.

A medical scan analysis function entry 356 can include functionclassifier data 610. Function classifier data 610 can include input andoutput types corresponding to the function. For example the functionclassifier data can include input scan category 611 that indicates whichtypes of scans can be used as input to the medical scan analysisfunction. For example, input scan category 611 can indicate that amedical scan analysis function is for chest CT scans from a particularhospital or other medical entity. The input scan category 611 caninclude one or more categories included in scan classifier data 420. Invarious embodiments, the input scan category 611 corresponds to thetypes of medical scans that were used to train the medical scan analysisfunction. Function classifier data 610 can also include output type data612 that characterizes the type of output that will be produced by thefunction, for example, indicating that a medical scan analysis functionis used to generate medical codes 447. The input scan category 611 canalso include information identifying which subsystems 101 areresponsible for running the medical scan analysis function.

A medical scan analysis function entry 356 can include trainingparameters 620. This can include training set data 621, which caninclude identifiers for the data used to train the medical scan analysisfunction, such as a set of medical scan identifiers 353 corresponding tothe medical scans used to train the medical scan analysis function, alist of medical scan reports and corresponding medical codes used totrain the medical scan analysis function, etc. Alternatively or inaddition to identifying particular scans of the training set, thetraining set data 621 can identify training set criteria, such asnecessary scan classifier data 420, necessary abnormality locations,classifiers, or other criteria corresponding to abnormality annotationdata 442, necessary confidence score data 460, for example, indicatingthat only medical scans with diagnosis data 440 assigned a truth flag461 or with confidence score data 460 otherwise comparing favorably to atraining set confidence score threshold are included, a number ofmedical scans to be included and proportion data corresponding todifferent criteria, or other criteria used to populate a training setwith data of medical scans. Training parameters 620 can include modeltype data 622 indicating one or more types of model, methods, and/ortraining functions used to determine the medical scan analysis functionby utilizing the training set 621. Training parameters 620 can includemodel parameter data 623 that can include a set of features of thetraining data selected to train the medical scan analysis function,determined values for weights corresponding to selected input and outputfeatures, determined values for model parameters corresponding to themodel itself, etc. The training parameter data can also include testingdata 624, which can identify a test set of medical scans or other dataused to test the medical scan analysis function. The test set can be asubset of training set 621, include completely separate data thantraining set 621, and/or overlap with training set 621. Alternatively orin addition, testing data 624 can include validation parameters such asa percentage of data that will be randomly or pseudo-randomly selectedfrom the training set for testing, parameters characterizing a crossvalidation process, or other information regarding testing. Trainingparameters 620 can also include training error data 625 that indicates atraining error associated with the medical scan analysis function, forexample, based on applying cross validation indicated in testing data624.

A medical scan analysis function entry 356 can include performance scoredata 630. Performance data can include model accuracy data 631, forexample, generated and/or updated based on the accuracy of the functionwhen performed on new data. For example, the model accuracy data 631 caninclude or be calculated based on the model error for determined forindividual uses, for example, generated by comparing the output of themedical scan analysis function to corresponding data generated by userinput to interactive interface 275 in conjunction with a subsystem 101and/or generated by comparing the output of the medical scan analysisfunction to medical scans with a truth flag 461. The model accuracy data631 can include aggregate model accuracy data computed based on modelerror of individual uses of the function over time. The performancescore data 630 can also include model efficiency data 632, which can begenerated based on how quickly the medical scan analysis functionperforms, how much memory is utilized by medical scan analysis function,or other efficiency data relating to the medical scan analysis function.Some or all of the performance score data 630 can be based on trainingerror data 625 or other accuracy and/or efficiency data determinedduring training and/or validation. As used herein, a “high” performancescore refers to a more favorable performance or rating than a “low”performance score.

A medical scan analysis function entry 356 can include version data 640.The version data can include a version identifier 641. The version datacan indicate one or more previous version identifiers 642, which can mapto version identifiers 641 stored in other medical scan analysisfunction entry 356 that correspond to previous versions of the function.Alternatively or in addition, the version data can indicate multipleversions of the same type based on function classifier data 610, canindicate the corresponding order and/or rank of the versions, and/or canindicate training parameters 620 associated with each version.

A medical scan analysis function entry 356 can include remediation data650. Remediation data 650 can include remediation instruction data 651which can indicate the steps in a remediation process indicating how amedical scan analysis function is taken out of commission and/orreverted to a previous version in the case that remediation isnecessary. The version data 640 can further include remediation criteriadata 652, which can include threshold data or other criteria used toautomatically determine when remediation is necessary. For example, theremediation criteria data 652 can indicate that remediation is necessaryat any time where the model accuracy data and/or the model efficiencydata compares unfavorably to an indicated model accuracy thresholdand/or indicated model efficiency threshold. The remediation data 650can also include recommissioning instruction data 653, identifyingrequired criteria for recommissioning a medical scan analysis functionand/or updating a medical scan analysis function. The remediation data650 can also include remediation history, indicating one or moreinstances that the medical scan analysis function was taken out ofcommission and/or was recommissioned.

FIGS. 6A and 6B present an embodiment of a medical scan diagnosingsystem 108.

The medical scan diagnosing system 108 can generate inference data 1110for medical scans by utilizing a set of medical scan inference functions1105, stored and run locally, stored and run by another subsystem 101,and/or stored in the medical scan analysis function database 346, wherethe function and/or parameters of the function can be retrieved from thedatabase by the medical scan diagnosing system. For example, the set ofmedical scan inference function 1105 can include some or all medicalscan analysis functions described herein or other functions thatgenerate inference data 1110 based on some or all data corresponding toa medical scan such as some or all data of a medical scan entry 352.Each medical scan inference function 1105 in the set can correspond to ascan category 1120, and can be trained on a set of medical scans thatcompare favorably to the scan category 1120. For example, each inferencefunction can be trained on a set of medical scans of the one or moresame scan classifier data 420, such as the same and/or similar scantypes, same and/or similar anatomical regions locations, same and/orsimilar machine models, same and/or similar machine calibration, sameand/or similar contrasting agent used, same and/or similar originatingentity, same and/or similar geographical region, and/or otherclassifiers. Thus, the scan categories 1120 can correspond to one ormore of a scan type, scan anatomical region data, hospital or otheroriginating entity data, machine model data, machine calibration data,contrast agent data, geographic region data, and/or other scanclassifying data 420. For example, a first medical scan inferencefunction can be directed to characterizing knee x-rays, and a secondmedical scan inference function can be directed to chest CT scans. Asanother example, a first medical scan inference function can be directedto characterizing CT scans from a first hospital, and a second medicalscan image analysis function can be directed to characterizing CT scansfrom a second hospital.

Training on these categorized sets separately can ensure each medicalscan inference function 1105 is calibrated according to its scancategory 1120, for example, allowing different inference functions to becalibrated on type specific, anatomical region specific, hospitalspecific, machine model specific, and/or region-specific tendenciesand/or discrepancies. Some or all of the medical scan inferencefunctions 1105 can be trained by the medical scan image analysis systemand/or the medical scan natural language processing system, and/or somemedical scan inference functions 1105 can utilize both image analysisand natural language analysis techniques to generate inference data1110. For example, some or all of the inference functions can utilizeimage analysis of the medical scan image data 410 and/or naturallanguage data extracted from abnormality annotation data 442 and/orreport data 449 as input, and generate diagnosis data 440 such asmedical codes 447 as output. Each medical scan inference function canutilize the same or different learning models to train on the same ordifferent features of the medical scan data, with the same or differentmodel parameters, for example indicated in the model type data 622 andmodel parameter data 623. Model type and/or parameters can be selectedfor a particular medical scan inference function based on particularcharacteristics of the one or more corresponding scan categories 1120,and some or all of the indicated in the model type data 622 and modelparameter data 623 can be selected automatically by a subsystem duringthe training process based on the particular learned and/or otherwisedetermined characteristics of the one or more corresponding scancategories 1120.

As shown in FIG. 6A, the medical scan diagnosing system 108 canautomatically select a medical scan for processing in response toreceiving it from a medical entity via the network. Alternatively, themedical scan diagnosing system 108 can automatically retrieve a medicalscan from the medical scan database that is selected based on a requestreceived from a user for a particular scan and/or based on a queue ofscans automatically ordered by the medical scan diagnosing system 108 oranother subsystem based on scan priority data 427.

Once a medical scan to be processed is determined, the medical scandiagnosing system 108 can automatically select an inference function1105 based on a determined scan category 1120 of the selected medicalscan and based on corresponding inference function scan categories. Thescan category 1120 of a scan can be determined based one some or all ofthe scan classifier data 420 and/or based on other metadata associatedwith the scan. This can include determining which one of the pluralityof medical scan inference functions 1105 matches or otherwise comparesfavorably to the scan category 1120, for example, by comparing the scancategory 1120 to the input scan category of the function classifier data610.

Alternatively or in addition, the medical scan diagnosing system 108 canautomatically determine which medical scan inference function 1105 isutilized based on an output preference that corresponding to a desiredtype of inference data 1110 that is outputted by an inference function1105. The output preference designated by a user of the medical scandiagnosing system 108 and/or based on the function of a subsystem 101utilizing the medical scan diagnosing system 108. For example, the setof inference functions 1105 can include inference functions that areutilized to indicate whether or not a medical scan is normal, toautomatically identify at least one abnormality in the scan, toautomatically characterize the at least one abnormality in the scan, toassign one or more medical codes to the scan, to generate naturallanguage text data and/or a formatted report for the scan, and/or toautomatically generate other diagnosis data such as some or all ofdiagnosis data 440 based on the medical scan. Alternatively or inaddition, some inference functions can also be utilized to automaticallygenerate confidence score data 460, display parameter data 470, and/orsimilar scan data 480. The medical scan diagnosing system 108 cancompare the output preference to the output type data 612 of the medicalscan inference function 1105 to determine the selected inferencefunction 1105. For example, this can be used to decide between a firstmedical scan inference function that automatically generates medicalcodes and a second medical scan inference function that automaticallygenerates natural language text for medical reports based on the desiredtype of inference data 1110.

Prior to performing the selected medical scan inference function 1105,the medical scan diagnosing system 108 can automatically perform aninput quality assurance function 1106 to ensure the scan classifier data420 or other metadata of the medical scan accurately classifies themedical scan such that the appropriate medical scan inference function1105 of the appropriate scan category 1120 is selected. The inputquality assurance function can be trained on, for example, medical scanimage data 410 of plurality of previous medical scans with verified scancategories. Thus, the input quality assurance function 1106 can takemedical scan image data 410 as input and can generate an inferred scancategory as output. The inferred scan category can be compared to thescan category 1120 of the scan, and the input quality assurance function1106 can determine whether or not the scan category 1120 is appropriateby determining whether the scan category 1120 compares favorably to theautomatically generated inferred scan category. The input qualityassurance function 1106 can also be utilized to reassign the generatedinferred scan category to the scan category 1120 when the scan category1120 compares favorably to the automatically generated inferred scancategory. The input quality assurance function 1106 can also be utilizedto assign the generated inferred scan category to the scan category 1120for incoming medical scans that do not include any classifying data,and/or to add classifiers in scan classifier data 420 to medical scansmissing one or more classifiers.

In various embodiments, upon utilizing the input quality assurancefunction 1106 to determine that the scan category 1120 determined by ascan classifier data 420 or other metadata is inaccurate, the medicalscan diagnosing system 108 can transmit an alert and/or an automaticallygenerated inferred scan category to the medical entity indicating thatthe scan is incorrectly classified in the scan classifier data 420 orother metadata. In some embodiments, the medical scan diagnosing system108 can automatically update performance score data corresponding to theoriginating entity of the scan indicated in originating entity data 423,or another user or entity responsible for classifying the scan, forexample, where a lower performance score is generated in response todetermining that the scan was incorrectly classified and/or where ahigher performance score is generated in response to determining thatthe scan was correctly classified.

In some embodiments, the medical scan diagnosing system 108 can transmitthe medical scan and/or the automatically generated inferred scancategory to a selected user. The user can be presented the medical scanimage data 410 and/or other data of the medical scan via the interactiveinterface 275, for example, displayed in conjunction with the medicalscan assisted review system 102. The interface can prompt the user toindicate the appropriate scan category 1120 and/or prompt the user toconfirm and/or edit the inferred scan category, also presented to theuser. For example, scan review data can be automatically generated toreflect the user generated and/or verified scan category 1120, This userindicated scan category 1120 can be utilized to select to the medicalscan inference function 1105 and/or to update the scan classifier data420 or other metadata accordingly. In some embodiments, for example,where the scan review data indicates that the selected user disagreeswith the automatically generated inferred scan category created by theinput quality assurance function 1106, the medical scan diagnosingsystem 108 can automatically update performance score data 630 of theinput quality assurance function 1106 by generating a low performancescore and/or determine to enter the remediation step 1140 for the inputquality assurance function 1106.

The medical scan diagnosing system 108 can also automatically perform anoutput quality assurance step after a medical scan inference function1105 has been performed on a medical scan to produce the inference data1110, as illustrated in the embodiment presented in FIG. 6B. The outputquality assurance step can be utilized to ensure that the selectedmedical scan inference function 1105 generated appropriate inferencedata 1110 based on expert feedback. The inference data 1110 generated byperforming the selected medical scan inference function 1105 can be sentto a client device 120 of a selected expert user, such as an expert userin the user database selected based on categorized performance dataand/or qualification data that corresponds to the scan category 1120and/or the inference itself, for example, by selecting an expert userbest suited to review an identified abnormality classifier category 444and/or abnormality pattern category 446 in the inference data 1110 basedon categorized performance data and/or qualification data of acorresponding user entry. The selected user can also correspond to amedical professional or other user employed at the originating entityand/or corresponding to the originating medical professional, indicatedin the originating entity data 423.

FIG. 6B illustrates an embodiment of the medical scan diagnosing system108 in conjunction with performing a remediation step 1140. The medicalscan diagnosing system 108 can monitor the performance of the set ofmedical scan inference functions 1105, for example, based on evaluatinginference accuracy data outputted by an inference data evaluationfunction and/or based monitoring on the performance score data 630 inthe medical scan analysis function database, and can determine whetheror not if the corresponding medical scan inference function 1105 isperforming properly. This can include, for example, determining if aremediation step 1140 is necessary for a medical scan inference function1105, for example, by comparing the performance score data 630 and/orinference accuracy data to remediation criteria data 652. Determining ifa remediation step 1140 is necessary can also be based on receiving anindication from the expert user or another user that remediation isnecessary for one or more identified medical scan inference functions1105 and/or for all of the medical scan inference functions 1105.

In various embodiments, a remediation evaluation function is utilized todetermine if a remediation step 1140 is necessary for medical scaninference function 1105. The remediation evaluation function can includedetermining that remediation is necessary when recent accuracy dataand/or efficiency data of a particular medical scan inference function1105 is below the normal performance level of the particular inferencefunction. The remediation evaluation function can include determiningthat remediation is necessary when recent or overall accuracy dataand/or efficiency data of a particular medical scan inference function1105 is below a recent or overall average for all or similar medicalscan inference functions 1105. The remediation evaluation function caninclude determining that remediation is necessary only after a thresholdnumber of incorrect diagnoses are made. In various embodiments, multiplethreshold number of incorrect diagnoses correspond to differentdiagnoses categories. For example, the threshold number of incorrectdiagnoses for remediation can be higher for false negative diagnosesthan false positive diagnoses. Similarly, categories corresponding todifferent diagnosis severities and/or rarities can have differentthresholds, for example where a threshold number of more severe and/ormore rare diagnoses that were inaccurate to necessitate remediation islower than a threshold number of less severe and/or less rare diagnosesthat were inaccurate.

The remediation step 1140 can include automatically updating anidentified medical inference function 1105. This can includeautomatically retraining identified medical inference function 1105 onthe same training set or on a new training set that includes new data,data with higher corresponding confidence scores, or data selected basedon new training set criteria. The identified medical inference function1105 can also be updated and/or changed based on the review datareceived from the client device. For example, the medical scan andexpert feedback data can be added to the training set of the medicalscan inference function 1105, and the medical scan inference function1105 can be retrained on the updated training set. Alternatively or inaddition, the expert user can identify additional parameters and/orrules in the expert feedback data based on the errors made by theinference function in generating the inference data 1110 for the medicalscan, and these parameters and/or rules can be applied to update themedical scan inference function, for example, by updating the model typedata 622 and/or model parameter data 623.

The remediation step 1140 can also include determining to split a scancategory 1120 into two or more subcategories. Thus, two or more newmedical scan inference functions 1105 can be created, where each newmedical scan inference functions 1105 is trained on a correspondingtraining set that is a subset of the original training set and/orincludes new medical scan data corresponding to the subcategory. Thiscan allow medical scan inference functions 1105 to become morespecialized and/or allow functions to utilize characteristics and/ordiscrepancies specific to the subcategory when generating inference data1110. Similarly, a new scan category 1120 that was not previouslyrepresented by any of the medical scan inference functions 1105 can beadded in the remediation step, and a new medical scan inferencefunctions 1105 can be trained on a new set of medical scan data thatcorresponds to the new scan category 1120. Splitting a scan categoryand/or adding a scan category can be determined automatically by themedical scan diagnosing system 108 when performing the remediation step1140, for example, based on performance score data 630. This can also bedetermined based on receiving instructions to split a category and/oradd a new scan category from the expert user or other user of thesystem.

After a medical scan inference function 1105 is updated or created forthe first time, the remediation step 1140 can further undergo acommissioning test, which can include rigorous testing of the medicalscan inference function 1105 on a testing set, for example, based on thetraining parameters 620. For example, the commissioning test can bepassed when the medical scan inference function 1105 generates athreshold number of correct inference data 1110 and/or the test can bepassed if an overall or average discrepancy level between the inferencedata and the test data is below a set error threshold. The commissioningtest can also evaluate efficiency, where the medical scan inferencefunction 1105 only passes the commissioning test if it performs at orexceeds a threshold efficiency level. If the medical scan inferencefunction 1105 fails the commissioning test, the model type and/or modelparameters can be modified automatically or based on user input, and themedical scan inference function can be retested, continuing this processuntil the medical scan inference function 1105 passes the commissioningtest.

The remediation step 1140 can include decommissioning the medical scaninference function 1105, for example, while the medical scan inferencefunction is being retrained and/or is undergoing the commissioning test.Incoming scans to the medical scan diagnosing system 108 with a scancategory 1120 corresponding to a decommissioned medical scan inferencefunction 1105 can be sent directly to review by one or more users, forexample, in conjunction with the medical scan annotator system 106.These user-reviewed medical scans and corresponding annotations can beincluded in an updated training set used to train the decommissionedmedical scan inference function 1105 as part of the remediation step1140. In some embodiments, previous versions of the plurality of medicalscan image analysis functions can be stored in memory of the medicalscan diagnosing system and/or can be determined based on the versiondata 640 of a medical scan inference function 1105. A previous versionof a medical scan inference function 1105, such as most recent versionor version with the highest performance score, can be utilized duringthe remediation step 1140 as an alternative to sending all medical scansto user review.

A medical scan inference function can also undergo the remediation step1140 automatically in response to a hardware and/or software update onprocessing, memory, and/or other computing devices where the medicalscan inference function 1105 is stored and/or performed. Differentmedical scan inference functions 1105 can be containerized on their owndevices by utilizing a micro-service architecture, so hardware and/orsoftware updates may only necessitate that one of the medical scaninference functions 1105 undergo the remediation step 1140 while theothers remain unaffected. A medical scan inference function 1105 canalso undergo the remediation step 1140 automatically in response tonormal system boot-up, and/or periodically in fixed intervals. Forexample, in response to a scheduled or automatically detected hardwareand/or software update, change, or issue, one or more medical scaninference functions 1105 affected by this hardware or software can betaken out of commission until they each pass the commissioning test.Such criteria can be indicated in the remediation criteria data 652.

The medical scan diagnosing system 108 can automatically manage usagedata, subscription data, and/or billing data for the plurality of userscorresponding to user usage of the system, for example, by utilizing,generating, and/or updating some or all of the subscription data of theuser database. Users can pay for subscriptions to the system, which caninclude different subscription levels that can correspond to differentcosts. For example, a hospital can pay a monthly cost to automaticallydiagnose up to 100 medical scans per month. The hospital can choose toupgrade their subscription or pay per-scan costs for automaticdiagnosing of additional scans received after the quota is reachedand/or the medical scan diagnosing system 108 can automatically sendmedical scans received after the quota is reached to an expert userassociated with the hospital. In various embodiments incentive programscan be used by the medical scan diagnosing system to encourage expertsto review medical scans from different medical entities. For example, anexpert can receive credit to their account and/or subscription upgradesfor every medical scan reviewed, or after a threshold number of medicalscans are reviewed. The incentive programs can include interactions by auser with other subsystems, for example, based on contributions made tomedical scan entries via interaction with other subsystems.

FIG. 7A presents an embodiment of a medical scan image analysis system112. A training set of medical scans used to train one more medical scanimage analysis functions can be received from one or more client devicesvia the network and/or can be retrieved from the medical scan database342, for example, based on training set data 621 corresponding tomedical scan image analysis functions. Training set criteria, forexample, identified in training parameters 620 of the medical scan imageanalysis function, can be utilized to automatically identify and selectmedical scans to be included in the training set from a plurality ofavailable medical scans. The training set criteria can be automaticallygenerated based on, for example, previously learned criteria, and/ortraining set criteria can be received via the network, for example, froman administrator of the medical scan image analysis system. The trainingset criteria can include a minimum training set size. The training setcriteria can include data integrity requirements for medical scans inthe training set such as requiring that the medical scan is assigned atruth flag 461, requiring that performance score data for a hospitaland/or medical professional associated with the medical scan comparesfavorably to a performance score threshold, requiring that the medicalscan has been reviewed by at least a threshold number of medicalprofessionals, requiring that the medical scan and/or a diagnosiscorresponding to a patient file of the medical scan is older than athreshold elapsed time period, or based on other criteria intended toinsure that the medical scans and associated data in the training set isreliable enough to be considered “truth” data. The training set criteriacan include longitudinal requirements such the number of requiredsubsequent medical scans for the patient, multiple required types ofadditional scans for the patient, and/or other patient filerequirements.

The training set criteria can include quota and/or proportionrequirements for one or more medical scan classification data. Forexample, the training set criteria can include meeting quota and/orproportion requirements for one or more scan types and/or human bodylocation of scans, meeting quota or proportion requirements for a numberof normal medical scans and a number of medicals scans with identifiedabnormalities, meeting quota and/or proportion requirements for a numberof medical scans with abnormalities in certain locations and/or a numberof medical scans with abnormalities that meet certain size, type, orother characteristics, meeting quota and/or proportion data for a numberof medical scans with certain diagnosis or certain corresponding medicalcodes, and/or meeting other identified quota and/or proportion datarelating to metadata, patient data, or other data associated with themedical scans.

In some embodiments, multiple training sets are created to generatecorresponding medical scan image analysis functions, for example,corresponding to some or all of the set of medical scan inferencefunctions 1105. Some or all training sets can be categorized based onsome or all of the scan classifier data 420 as described in conjunctionwith the medical scan diagnosing system 108, where medical scans areincluded in a training set based on their scan classifier data 420matching the scan category of the training set. In some embodiments, theinput quality assurance function 1106 or another input check step can beperformed on medical scans selected for each training set to confirmthat their corresponding scan classifier data 420 is correct. In someembodiments, the input quality assurance function can correspond to itsown medical scan image analysis function, trained by the medical scanimage analysis system, where the input quality assurance functionutilizes high level computer vision technology to determine a scancategory 1120 and/or to confirm the scan classifier data 420 alreadyassigned to the medical scan.

In some embodiments, the training set will be used to create a singleneural network model, or other model corresponding to model type data622 and/or model parameter data 623 of the medical scan image analysisfunction that can be trained on some or all of the medical scanclassification data described above and/or other metadata, patient data,or other data associated with the medical scans. In other embodiments, aplurality of training sets will be created to generate a plurality ofcorresponding neural network models, where the multiple training setsare divided based on some or all of the medical scan classification datadescribed above and/or other metadata, patient data, or other dataassociated with the medical scans. Each of the plurality of neuralnetwork models can be generated based on the same or different learningalgorithm that utilizes the same or different features of the medicalscans in the corresponding one of the plurality of training sets. Themedical scan classifications selected to segregate the medical scansinto multiple training sets can be received via the network, for examplebased on input to an administrator client device from an administrator.The medical scan classifications selected to segregate the medical scanscan be automatically determined by the medical scan image analysissystem, for example, where an unsupervised clustering algorithm isapplied to the original training set to determine appropriate medicalscan classifications based on the output of the unsupervised clusteringalgorithm.

In embodiments where the medical scan image analysis system is used inconjunction with the medical scan diagnosing system, each of the medicalscan image analysis functions associated with each neural network modelcan correspond to one of the plurality of neural network modelsgenerated by the medical scan image analysis system. For example, eachof the plurality of neural network models can be trained on a trainingset classified on scan type, scan human body location, hospital or otheroriginating entity data, machine model data, machine calibration data,contrast agent data, geographic region data, and/or other scanclassifying data as discussed in conjunction with the medical scandiagnosing system. In embodiments where the training set classifiers arelearned, the medical scan diagnosing system can determine which of themedical scan image analysis functions should be applied based on thelearned classifying criteria used to segregate the original trainingset.

A computer vision-based learning algorithm used to create each neuralnetwork model can include selecting a three-dimensional subregion 1310for each medical scan in the training set. This three-dimensionalsubregion 1310 can correspond to a region that is “sampled” from theentire scan that may represent a small fraction of the entire scan.Recall that a medical scan can include a plurality of orderedcross-sectional image slices. Selecting a three-dimensional subregion1310 can be accomplished by selecting a proper image slice subset 1320of the plurality of cross-sectional image slices from each of theplurality of medical scans, and by further selecting a two-dimensionalsubregion 1330 from each of the selected subset of cross-sectional imageslices of the each of the medical scans. In some embodiments, theselected image slices can include one or more non-consecutive imageslices and thus a plurality of disconnected three-dimensional subregionswill be created. In other embodiments, the selected proper subset of theplurality of image slices correspond to a set of consecutive imageslices, as to ensure that a single, connected three-dimensionalsubregion is selected. In some embodiments, entire scans of the trainingset are used to train the neural network model. In such embodiment, asused herein, the three-dimensional subregion 1310 can refer to all ofthe medical scan image data 410 of a medical scan.

In some embodiments, a density windowing step can be applied to the fullscan or the selected three-dimensional subregion. The density windowingstep can include utilizing a selected upper density value cut off and/ora selected lower density value cut off, and masking pixels with highervalues than the upper density value cut off and/or masking pixels withlower values than the lower density value cut off. The upper densityvalue cut off and/or a selected lower density value cut off can bedetermined based on based on the range and/or distribution of densityvalues included in the region that includes the abnormality, and/orbased on the range and/or distribution of density values associated withthe abnormality itself, based on user input to a subsystem, based ondisplay parameter data associated with the medical scan or associatedwith medical scans of the same type, and/or can be learned in thetraining step. In some embodiments, a non-linear density windowingfunction can be applied to alter the pixel density values, for example,to stretch or compress contrast. In some embodiments, this densitywindowing step can be performed as a data augmenting step, to createadditional training data for a medical scan in accordance with differentdensity windows.

Having determined the subregion training set 1315 of three-dimensionalsubregions 1310 corresponding to the set of full medical scans in thetraining set, the medical scan image analysis system can complete atraining step 1352 by performing a learning algorithm on the pluralityof three-dimensional subregions to generate model parameter data 1355 ofa corresponding learning model. The learning model can include one ormore of a neural network, an artificial neural network, a convolutionalneural network, a Bayesian model, a support vector machine model, acluster analysis model, or other supervised or unsupervised learningmodel. The model parameter data 1355 can generated by performing thelearning algorithm 1350, and the model parameter data 1355 can beutilized to determine the corresponding medical scan image analysisfunctions. For example, some or all of the model parameter data 1355 canbe mapped to the medical scan analysis function in the model parameterdata 623 or can otherwise define the medical scan analysis function.

The training step 1352 can include creating feature vectors for eachthree-dimensional subregion of the training set for use by the learningalgorithm 1350 to generate the model parameter data 1355. The featurevectors can include the pixel data of the three-dimensional subregionssuch as density values and/or grayscale values of each pixel based on adetermined density window. The feature vectors can also include otherfeatures as additional input features or desired output features, suchas known abnormality data such as location and/or classification data,patient history data such as risk factor data or previous medical scans,diagnosis data, responsible medical entity data, scan machinery model orcalibration data, contrast agent data, medical code data, annotationdata that can include raw or processed natural language text data, scantype and/or anatomical region data, or other data associated with theimage, such as some or all data of a medical scan entry 352. Featurescan be selected based on administrator instructions received via thenetwork and/or can be determined based on determining a feature set thatreduces error in classifying error, for example, by performing across-validation step on multiple models created using different featuresets. The feature vector can be split into an input feature vector andoutput feature vector. The input feature vector can include data thatwill be available in subsequent medical scan input, which can includefor example, the three-dimensional subregion pixel data and/or patienthistory data. The output feature vector can include data that will beinferred in in subsequent medical scan input and can include singleoutput value, such as a binary value indicating whether or not themedical scan include an abnormality or a value corresponding to one of aplurality of medical codes corresponding to the image. The outputfeature vector can also include multiple values which can includeabnormality location and/or classification data, diagnosis data, orother output. The output feature vector can also include a determinedupper density value cut off and/or lower density value cut off, forexample, characterizing which pixel values were relevant to detectingand/or classifying an abnormality. Features included in the outputfeature vector can be selected to include features that are known in thetraining set, but may not be known in subsequent medical scans such astriaged scans to be diagnosed by the medical scan diagnosing system,and/or scans to be labeled by the medical scan report labeling system.The set of features in the input feature vector and output featurevector, as well as the importance of different features where eachfeature is assigned a corresponding weight, can also be designated inthe model parameter data 1355.

Consider a medical scan image analysis function that utilizes a neuralnetwork. The neural network can include a plurality of layers, whereeach layer includes a plurality of neural nodes. Each node in one layercan have a connection to some or all nodes in the next layer, where eachconnection is defined by a weight value. Thus, the model parameter data1355 can include a weight vector that includes weight values for everyconnection in the network. Alternatively or in addition, the modelparameter data 1355 can include any vector or set of parametersassociated with the neural network model, which can include an upperdensity value cut off and/or lower density value cut off used to masksome of the pixel data of an incoming image, kernel values, filterparameters, bias parameters, and/or parameters characterizing one ormore of a plurality of convolution functions of the neural networkmodel. The medical scan image analysis function can be utilized toproduce the output vector as a function of the input feature vector andthe model parameter data 1355 that characterizes the neural networkmodel. In particular, the medical scan image analysis function caninclude performing a forward propagation step plurality of neuralnetwork layers to produce an inferred output vector based on the weightvector or other model parameter data 1355. Thus, the learning algorithm1350 utilized in conjunction with a neural network model can includedetermining the model parameter data 1355 corresponding to the neuralnetwork model, for example, by populating the weight vector with optimalweights that best reduce output error.

In particular, determining the model parameter data 1355 can includeutilizing a backpropagation strategy. The forward propagation algorithmcan be performed on at least one input feature vector corresponding toat least one medical scan in the training set to propagate the at leastone input feature vector through the plurality of neural network layersbased on initial and/or default model parameter data 1355, such as aninitial weight vector of initial weight values set by an administratoror chosen at random. The at least one output vector generated byperforming the forward propagation algorithm on the at least one inputfeature vector can be compared to the corresponding at least one knownoutput feature vector to determine an output error. Determining theoutput error can include, for example, computing a vector distance suchas the Euclidian distance, or squared Euclidian distance, between theproduced output vector and the known output vector, and/or determiningan average output error such as an average Euclidian distance or squaredEuclidian distance if multiple input feature vectors were employed.Next, gradient descent can be performed to determine an updated weightvector based on the output error or average output error. This gradientdescent step can include computing partial derivatives for the errorwith respect to each weight, or other parameter in the model parameterdata 1355, at each layer starting with the output layer. Chain rule canbe utilized to iteratively compute the gradient with respect to eachweight or parameter at each previous layer until all weight's gradientsare computed. Next updated weights, or other parameters in the modelparameter data 1355, are generated by updating each weight based on itscorresponding calculated gradient. This process can be repeated on atleast one input feature vector, which can include the same or differentat least one feature vector used in the previous iteration, based on theupdated weight vector and/or other updated parameters in the modelparameter data 1355 to create a new updated weight vector and/or othernew updated parameters in the model parameter data 1355. This processcan continue to repeat until the output error converges, the outputerror is within a certain error threshold, or another criterion isreached to determine the most recently updated weight vector and/orother model parameter data 1355 is optimal or otherwise determined forselection.

Having determined the medical scan neural network and its final othermodel parameter data 1355, an inference step 1354 can be performed onnew medical scans to produce inference data 1370, such as inferredoutput vectors, as shown in FIG. 7B. The inference step can includeperforming the forward propagation algorithm to propagate an inputfeature vector through a plurality of neural network layers based on thefinal model parameter data 1355, such as the weight values of the finalweight vector, to produce the inference data. This inference step 1354can correspond to performing the medical scan image analysis function,as defined by the final model parameter data 1355, on new medical scansto generate the inference data 1370, for example, in conjunction withthe medical scan diagnosing system 108 to generate inferred diagnosisdata or other selected output data for triaged medical scans based onits corresponding the input feature vector.

The inference step 1354 can include applying the density windowing stepto new medical scans. Density window cut off values and/or a non-lineardensity windowing function that are learned can be automatically appliedwhen performing the inference step. For example, if the training step1352 was used to determine optimal upper density value cut off and/orlower density value cut off values to designate an optimal densitywindow, the inference step 1354 can include masking pixels of incomingscans that fall outside of this determined density window beforeapplying the forward propagation algorithm. As another example, iflearned parameters of one or more convolutional functions correspond tothe optimal upper density value cut off and/or lower density value cutoff values, the density windowing step is inherently applied when theforward propagation algorithm is performed on the new medical scans.

In some embodiments where a medical scan analysis function is defined bymodel parameter data 1355 corresponding to a neutral network model, theneural network model can be a fully convolutional neural network. Insuch embodiments, only convolution functions are performed to propagatethe input feature vector through the layers of the neural network in theforward propagation algorithm. This enables the medical scan imageanalysis functions to process input feature vectors of any size. Forexample, as discussed herein, the pixel data corresponding to thethree-dimensional subregions is utilized input to the forwardpropagation algorithm when the training step 1352 is employed topopulate the weight vector and/or other model parameter data 1355.However, when performing the forward propagation algorithm in theinference step 1354, the pixel data of full medical scans can beutilized as input, allowing the entire scan to be processed to detectand/or classify abnormalities, or otherwise generate the inference data1370. This may be a preferred embodiment over other embodiments wherenew scans must also be sampled by selecting a three-dimensionalsubregions and/or other embodiments where the inference step requires“piecing together” inference data 1370 corresponding to multiplethree-dimensional subregions processed separately.

The inferred output vector of the inference data 1370 can include aplurality of abnormality probabilities mapped to a pixel location ofeach of a plurality of cross-sectional image slices of the new medicalscan. For example, the inferred output vector can indicate a set ofprobability matrices 1371, where each matrix in the set corresponds toone of the plurality of image slices of the medical scan, where eachmatrix is a size corresponding to the number of pixels in each imageslice, where each cell of each matrix corresponds to a pixel of thecorresponding image slice, whose value is the abnormality probability ofthe corresponding pixel.

A detection step 1372 can include determining if an abnormality ispresent in the medical scan based on the plurality of abnormalityprobabilities. Determining if an abnormality is present can include, forexample, determining that a cluster of pixels in the same region of themedical scan correspond to high abnormality probabilities, for example,where a threshold proportion of abnormality probabilities must meet orexceed a threshold abnormality probability, where an average abnormalityprobability of pixels in the region must meet or exceed a thresholdabnormality probability, where the region that includes the cluster ofpixels must be at least a certain size, etc. Determining if anabnormality is present can also include calculating a confidence scorebased on the abnormality probabilities and/or other data correspondingto the medical scan such as patient history data. The location of thedetected abnormality can be determined in the detection step 1372 basedon the location of the pixels with the high abnormality probabilities.The detection step can further include determining an abnormality region1373, such as a two-dimensional subregion on one or more image slicesthat includes some or all of the abnormality. The abnormality region1373 determined in the detection step 1372 can be mapped to the medicalscan to populate some or all of the abnormality location data 443 foruse by one or more other subsystems 101 and/or client devices 120.Furthermore, determining whether or not an abnormality exists in thedetection step 1372 can be used to populate some or all of the diagnosisdata 440 of the medical scan, for example, to indicate that the scan isnormal or contains an abnormality in the diagnosis data 440.

An abnormality classification step 1374 can be performed on a medicalscan in response to determining an abnormality is present.Classification data 1375 corresponding to one or more classificationcategories such as abnormality size, volume, pre-post contract, doublingtime, calcification, components, smoothness, texture, diagnosis data,one or more medical codes, a malignancy rating such as a Lung-RADSscore, or other classifying data as described herein can be determinedbased on the detected abnormality. The classification data 1375generated by the abnormality classification step 1374 can be mapped tothe medical scan to populate some or all of the abnormalityclassification data 445 of the corresponding abnormality classifiercategories 444 and/or abnormality pattern categories 446 and/or todetermine one or more medical codes 447 of the medical scan. Theabnormality classification step 1374 can include performing anabnormality classification function on the full medical scan, or theabnormality region 1373 determined in the detection step 1372. Theabnormality classification function can be based on another modeltrained on abnormality data such as a support vector machine model,another neural network model, or any supervised classification modeltrained on medical scans, or portions of medical scans, that includeknown abnormality classifying data to generate inference data for someor all of the classification categories. For example, the abnormalityclassification function can include another medical scan analysisfunction. Classification data 1375 in each of a plurality ofclassification categories can also be assigned their own calculatedconfidence score, which can also be generated by utilizing theabnormality classification function. Output to the abnormalityclassification function can also include at least one identified similarmedical scan and/or at least one identified similar cropped image, forexample, based on the training data. The abnormality classification stepcan also be included in the inference step 1354, where the inferredoutput vector or other inference data 1370 of the medical scan imageanalysis function includes the classification data 1375.

The abnormality classification function can be trained on full medicalscans and/or one or more cropped or full selected image slices frommedical scans that contain an abnormality. For example, the abnormalityclassification function can be trained on a set of two-dimensionalcropped slices that include abnormalities. The selected image slicesand/or the cropped region in each selected image slice for each scan inthe training set can be automatically selected based upon the knownlocation of the abnormality. Input to the abnormality classificationfunction can include the full medical scan, one or more selected fullimage slices, and/or one or more selected image slices cropped based ona selected region. Thus, the abnormality classification step can includeautomatically selecting one or more image slices that include thedetected abnormality. The slice selection can include selecting thecenter slice in a set of consecutive slices that are determined toinclude the abnormality or selecting a slice that has the largestcross-section of the abnormality, or selecting one or more slices basedon other criteria. The abnormality classification step can also includeautomatically generating one or more cropped two-dimensional imagescorresponding to the one or more of the selected image slices based onan automatically selected region that includes the abnormality.

Input to the abnormality classification function can also include otherdata associated with the medical scan, including patient history, riskfactors, or other metadata. The abnormality classification step can alsoinclude determining some or all of the characteristics based on data ofthe medical scan itself. For example, the abnormality size and volumecan be determined based on a number of pixels determined to be part ofthe detected abnormality. Other classifiers such as abnormality textureand/or smoothness can be determined by performing one or more otherpreprocessing functions on the image specifically designed tocharacterize such features. Such preprocessed characteristics can beincluded in the input to the abnormality classification function to themore difficult task of assigning a medical code or generating otherdiagnosis data. The training data can also be preprocessed to includesuch preprocessed features.

A similar scan identification step 1376 can also be performed on amedical scan with a detected abnormality and/or can be performed on theabnormality region 1373 determined in the detection step 1372. Thesimilar scan identification step 1376 can include generating similarabnormality data 1377, for example, by identifying one or more similarmedical scans or one or more similar cropped two-dimensional images froma database of medical scans and/or database of cropped two-dimensionalimages. Similar medical scans and/or cropped images can include medicalscans or cropped images that are visually similar, medical scans orcropped images that have known abnormalities in a similar location to aninferred abnormality location of the given medical scan, medical scansthat have known abnormalities with similar characteristics to inferredcharacteristics of an abnormality in the given scan, medical scans withsimilar patient history and/or similar risk factors, or some combinationof these factors and/or other known and/or inferred factors. The similarabnormality data 1377 can be mapped to the medical scan to populate someor all of its corresponding similar scan data 480 for use by one or moreother subsystems 101 and/or client devices 120.

The similar scans identification step 1376 can include performing a scansimilarity algorithm, which can include generating a feature vector forthe given medical scan and for medical scans in the set of medicalscans, where the feature vector can be generated based on quantitativeand/or category based visual features, inferred features, abnormalitylocation and/or characteristics such as the predetermined size and/orvolume, patient history and/or risk factor features, or other known orinferred features. A medical scan similarity analysis function can beapplied to the feature vector of the given medical scan and one or morefeature vectors of medical scans in the set. The medical scan similarityanalysis function can include computing a similarity distance such asthe Euclidian distance between the feature vectors, and assigning thesimilarity distance to the corresponding medical scan in the set.Similar medical scans can be identified based on determining one or moremedical scans in the set with a smallest computed similarity distance,based on ranking medical scans in the set based on the computedsimilarity distances and identifying a designated number of top rankedmedical scans, and/or based on determining if a similarity distancebetween the given medical scan and a medical scan in the set is smallerthan a similarity threshold. Similar medical scans can also beidentified based on determining medical scans in a database that mappedto a medical code that matches the medical code of the medical scan, ormapped to other matching classifying data. A set of identified similarmedical scans can also be filtered based on other inputted orautomatically generated criteria, where for example only medical scanswith reliable diagnosis data or rich patient reports, medical scans withcorresponding with longitudinal data in the patient file such asmultiple subsequent scans taken at later dates, medical scans withpatient data that corresponds to risk factors of the given patient, orother identified criteria, where only a subset of scans that comparefavorably to the criteria are selected from the set and/or only ahighest ranked single scan or subset of scans are selected from the set,where the ranking is automatically computed based on the criteria.Filtering the similar scans in this fashion can include calculating, orcan be based on previously calculated, one or more scores as discussedherein. For example, the ranking can be based on a longitudinal qualityscore, such as the longitudinal quality score 434, which can becalculated for an identified medical scan based on a number ofsubsequent and/or previous scans for the patient. Alternatively or inaddition, the ranking can be based on a confidence score associated withdiagnosis data of the scan, such as confidence score data 460, based onperformance score data associated with a user or medical entityassociated with the scan, based on an amount of patient history data ordata in the medical scan entry 352, or other quality factors. Theidentified similar medical scans can be filtered based on ranking thescans based on their quality score and/or based on comparing theirquality score to a quality score threshold. In some embodiments, alongitudinal threshold must be reached, and only scans that comparefavorably to the longitudinal threshold will be selected. For example,only scans with at least three scans on file for the patient and finalbiopsy data will be included.

In some embodiments, the similarity algorithm can be utilized inaddition to or instead of the trained abnormality classificationfunction to determine some or all of the inferred classification data1375 of the medical scan, based on the classification data such asabnormality classification data 445 or other diagnosis data 440 mappedto one or more of the identified similar scans. In other embodiments,the similarity algorithm is merely used to identify similar scans forreview by medical professionals to aid in review, diagnosis, and/orgenerating medical reports for the medical image.

A display parameter step 1378 can be performed based on the detectionand/or classification of the abnormality. The display parameter step caninclude generating display parameter data 1379, which can includeparameters that can be used by an interactive interface to best displayeach abnormality. The same or different display parameters can begenerated for each abnormality. The display parameter data generated inthe display parameter step 1378 can be mapped to the medical scan topopulate some or all of its corresponding display parameter data 470 foruse by one or more other subsystems 101 and/or client devices 120.

Performing the display parameter step 1378 can include selecting one ormore image slices that include the abnormality by determining the one ormore image slices that include the abnormality and/or determining one ormore image slices that has a most optimal two-dimensional view of theabnormality, for example by selecting the center slice in a set ofconsecutive slices that are determined to include the abnormality,selecting a slice that has the largest cross-section of the abnormality,selecting a slice that includes a two-dimensional image of theabnormality that is most similar to a selected most similartwo-dimensional-image, selecting the slice that was used as input to theabnormality classification step and/or similar scan identification step,or based on other criteria. This can also include automatically croppingone or more selected image slices based on an identified region thatincludes the abnormality. This can also select an ideal Hounsfieldwindow that best displays the abnormality. This can also includeselecting other display parameters based on data generated by themedical scan interface evaluating system and based on the medical scan.

FIGS. 8A-8F illustrate embodiments of a medical picture archiveintegration system 2600. The medical picture archive integration system2600 can provide integration support for a medical picture archivesystem 2620, such as a PACS that stores medical scans. The medicalpicture archive integration system 2600 can utilize model parametersreceived from a central server system 2640 via a network 2630 to performan inference function on de-identified medical scans of medical scansreceived from the medical picture archive system 2620. The annotationdata produced by performing the inference function can be transmittedback to the medical picture archive system. Furthermore, the annotationdata and/or de-identified medical scans can be sent to the centralserver system 2640, and the central server system can train on thisinformation to produce new and/or updated model parameters fortransmission back to the medical picture archive integration system 2600for use on subsequently received medical scans.

In various embodiments, medical picture archive integration system 2600includes a de-identification system that includes a first memorydesignated for protected health information (PHI), operable to perform ade-identification function on a DICOM image, received from a medicalpicture archive system, to identify at least one patient identifier andgenerate a de-identified medical scan that does not include the at leastone patient identifier. The medical picture archive integration systemfurther includes a de-identified image storage system that stores thede-identified medical scan in a second memory that is separate from thefirst memory, and an annotating system, operable to utilize modelparameters received from a central server to perform an inferencefunction on the de-identified medical scan, retrieved from the secondmemory to generate annotation data for transmission to the medicalpicture archive system as an annotated DICOM file.

The first memory and the second memory can be implemented by utilizingseparate storage systems: the first memory can be implemented by a firststorage system designated for PHI storage, and the second memory can beimplemented by a second storage system designated for storage ofde-identified data. The first storage system can be protected fromaccess by the annotating system, while the second storage system can beaccessible by the annotating system. The medical picture archiveintegration system 2600 can be operable to perform the de-identificationfunction on data in first storage system to generate de-identified data.The de-identified data can then be stored in the second storage systemfor access by the annotating system. The first and second storagesystems can be physically separate, each utilizing at least one of theirown, separate memory devices. Alternatively, the first and secondstorage systems can be virtually separate, where data is stored inseparate virtual memory locations on the same set of memory devices.Firewalls, virtual machines, and/or other protected containerization canbe utilized to enforce the separation of data in each storage system, toprotect the first storage system from access by the annotating systemand/or from other unauthorized access, and/or to ensure that only dataof the first storage system that has been properly de-identified throughapplication of the de-identification function can be stored in thesecond storage system.

As shown in FIG. 8A, the medical picture archive system 2620 can receiveimage data from a plurality of modality machines 2622, such as CTmachines, MRI machines, x-ray machines, and/or other medical imagingmachines that produce medical scans. The medical picture archive system2620 can store this image data in a DICOM image format and/or can storethe image data in a plurality of medical scan entries 352 as describedin conjunction with some or all of the attributes described inconjunction with FIGS. 4A and 4B. While “DICOM image” will be usedherein to refer to medical scans stored by the medical picture archivesystem 2620, the medical picture archive integration system 2600 canprovide integration support for medical picture archive systems 2620that store medical scans in other formats.

The medical picture archive integration system 2600 can include areceiver 2602 and a transmitter 2604, operable to transmit and receivedata from the medical picture archive system 2620, respectively. Forexample, the receiver 2602 and transmitter 2604 can be configured toreceive and transmit data, respectively, in accordance with a DICOMcommunication protocol and/or another communication protocol recognizedby the medical image archive system 2620. The receiver can receive DICOMimages from the medical picture archive system 2620. The transmitter2604 can send annotated DICOM files to the medical picture archivesystem 2620.

DICOM images received via receiver 2602 can be sent directly to ade-identification system 2608. The de-identification system 2608 can beoperable to perform a de-identification function on the first DICOMimage to identify at least one patient identifier in the DICOM image,and to generate a de-identified medical scan that does not include theidentified at least one patient identifier. As used herein, a patientidentifier can include any patient identifying data in the image data,header, and/or metadata of a medical scan, such as a patient ID numberor other unique patient identifier, an accession number, aservice-object pair (SOP) instance unique identifier (UID) field, scandate and/or time that can be used to determine the identity of thepatient that was scanned at that date and/or time, and/or other privatedata corresponding to the patient, doctor, or hospital. In someembodiments, the de-identified medical scan is still in a DICOM imageformat. For example, a duplicate DICOM image that does not include thepatient identifiers can be generated, and/or the original DICOM imagecan be altered such that the patient identifiers of the new DICOM imageare masked, obfuscated, removed, replaced with a custom fiducial, and/orotherwise anonymized. In other embodiments, the de-identified medicalscan is formatted in accordance with a different image format and/ordifferent data format that does not include the identifying information.In some embodiments, other private information, for example, associatedwith a particular doctor or other medical professional, can beidentified and anonymized as well.

Some patient identifying information can be included in a DICOM headerof the DICOM image, for example, in designated fields for patientidentifiers. These corresponding fields can be anonymized within thecorresponding DICOM header field. Other patient identifying informationcan be included in the image itself, such as in medical scan image data410. For example, the image data can include a patient name or otheridentifier that was handwritten on a hard copy of the image before theimage was digitized. As another example, a hospital administered armbandor other visual patient information in the vicinity of the patient mayhave been captured in the image itself. A computer vision model candetect the presence of these identifiers for anonymization, for example,where a new DICOM image includes a fiducial image that covers theidentifying portion of the original DICOM image. In some embodiments,patient information identified in the DICOM header can be utilized todetect corresponding patient information in the image itself. Forexample, a patient name extracted from the DICOM header beforeanonymization can be used to search for the patient name in the imageand/or to detect a location of the image that includes the patient name.In some embodiments, the de-identification system 2608 is implemented bythe de-identification system discussed in conjunction with FIGS. 10A,10B and 11 , and/or utilizes functions and/or operations discussed inconjunction with FIGS. 10A, 10B and 11 .

The de-identified medical scan can be stored in de-identified imagestorage system 2610 and the annotating system 2612 can access thede-identified medical scan from the de-identified image storage system2610 for processing. The de-identified storage system can archive aplurality of de-identified DICOM images and/or can serve as temporarystorage for the de-identified medical scan until processing of thede-identified medical scan by the annotating system 2612 is complete.The annotating system 2612 can generate annotation data by performing aninference function on the de-identified medical scan, utilizing themodel parameters received from the central server system 2640. Theannotation data can correspond to some or all of the diagnosis data 440as discussed in conjunction with FIGS. 4A and 4B. In come embodiments,the annotating system 2612 can utilize the model parameters to performinference step 1354, the detection step 1372, the abnormalityclassification step 1374, the similar scan identification step 1376,and/or the display parameter step 1378 of the medical scan imageanalysis system 112, as discussed in conjunction with FIG. 7B, onde-identified medical scans received from the medical picture archivesystem 2620.

In some embodiments, model parameters for a plurality of inferencefunctions can be received from the central server system 2640, forexample, where each inference function corresponds to one of a set ofdifferent scan categories. Each scan category can correspond to a uniquecombination of one or a plurality of scan modalities, one of a pluralityof anatomical regions, and/or other scan classifier data 420. Forexample, a first inference function can be trained on and intended forde-identified medical scans corresponding chest CT scans, and a secondinference function can be trained on and intended for de-identifiedmedical scans corresponding to head MRI scans. The annotating system canselect one of the set of inference functions based on determining thescan category of the DICOM image, indicated in the de-identified medicalscan, and selecting the inference function that corresponds to thedetermined scan category.

To ensure that scans received from the medical picture archive system2620 match the set of scan categories for which the annotating system isoperable to perform a corresponding inference function, the transmittercan transmit requests, such as DICOM queries, indicating image typeparameters such as parameters corresponding to scan classifier data 420,for example indicating one or more scan modalities, one or moreanatomical regions, and/or other parameters. For example, the requestcan indicate that all incoming scans that match the set of scancategories corresponding to a set of inference functions the annotatingsystem 2612 for which the annotating system has obtained modelparameters from the central server system 2640 and is operable toperform.

Once the annotation data is generated by performing the selectedinference function, the annotating system 2612 can generate an annotatedDICOM file for transmission to the medical image archive system 2620 forstorage. The annotated DICOM file can include some or all of the fieldsof the diagnosis data 440 and/or abnormality annotation data 442 ofFIGS. 4A and 4B. The annotated DICOM file can include scan overlay data,providing location data of an identified abnormality and/or display datathat can be used in conjunction with the original DICOM image toindicate the abnormality visually in the DICOM image and/or to otherwisevisually present the annotation data, for example, for use with themedical scan assisted review system 102. For example, a DICOMpresentation state file can be generated to indicate the location of anabnormality identified in the de-identified medical scan. The DICOMpresentation state file can include an identifier of the original DICOMimage, for example, in metadata of the DICOM presentation state file, tolink the annotation data to the original DICOM image. In otherembodiments, a full, duplicate DICOM image is generated that includesthe annotation data with an identifier linking this duplicate annotatedDICOM image to the original DICOM image.

The identifier linking the annotated DICOM file to the original DICOMimage can be extracted from the original DICOM file by thede-identification system 2608, thus enabling the medical picture archivesystem 2620 to link the annotated DICOM file to the original DICOM imagein its storage. For example, the de-identified medical scan can includean identifier that links the de-identified medical scan to the originalDICOM file, but does not link the de-identified medical scan to apatient identifier or other private data.

In some embodiments, generating the annotated DICOM file includesaltering one or more fields of the original DICOM header. For example,standardized header formatting function parameters can be received fromthe central server system and can be utilized by the annotating systemto alter the original DICOM header to match a standardized DICOM headerformat. The standardized header formatting function can be trained in asimilar fashion to other medical scan analysis functions discussedherein and/or can be characterized by some or all fields of a medicalscan analysis function entry 356. The annotating system can perform thestandardized header formatting function on a de-identified medical scanto generate a new, standardized DICOM header for the medical scan to besent back to the medical picture archive system 2620 in the annotatedDICOM file and/or to replace the header of the original DICOM file. Thestandardized header formatting function can be run in addition to otherinference functions utilized to generate annotation data. In otherembodiments, the medical picture archive integration system 2600 isimplemented primarily for header standardization for medical scansstored by the medical picture archive system 2620. In such embodiments,only the standardized header formatting function is performed on thede-identified data to generate a modified DICOM header for the originalDICOM image, but the de-identified medical scan is not annotated.

In some embodiments of header standardization, the annotation system canstore a set of acceptable, standardized entries for some or all of theDICOM header fields, and can select one of the set of acceptable,standardized entries in populating one or more fields of the new DICOMheader for the annotated DICOM file. For example, each of the set ofscan categories determined by the annotating system can correspond to astandardized entry of one or more fields of the DICOM header. The newDICOM header can thus be populated based on the determined scancategory.

In some embodiments, each of the set of standardized entries can bemapped to a set of related, non-standardized entries, such as entries ina different order, commonly misspelled entries, or other similar entriesthat do not follow a standardized format. For example, one of the set ofacceptable, standardized entries for a field corresponding to a scancategory can include “Chest CT”, which can be mapped to a set ofsimilar, non-standardized entries which can include “CT chest”,“computerized topography CT”, and/or other entries that are notstandardized. In such embodiments, the annotating system can determinethe original DICOM header is one of the similar non-standardizedentries, and can select the mapped, standardized entry as the entry forthe modified DICOM header. In other embodiments, the image data itselfand/or or other header data can be utilized by the annotation system todetermine a standardized field. For example, an input quality assurancefunction 1106 can be trained by the central server system and sent tothe annotating system to determine one or more appropriate scanclassifier fields, or one or more other DICOM header fields, based onthe image data or other data of the de-identified medical scan. One ormore standardized labels can be assigned to corresponding fields of themodified DICOM header based on the one or more fields determined by theinput quality assurance function.

In some embodiments, the DICOM header is modified based on theannotation data generated in performing the inference function. Inparticular, a DICOM priority header field can be generated and/ormodified automatically based on the severity and/or time-sensitivity ofthe abnormalities detected in performing the inference function. Forexample, a DICOM priority header field can be changed from a lowpriority to a high priority in response to annotation data indicating abrain bleed in the de-identified medical scan of a DICOM imagecorresponding to a head CT scan, and a new DICOM header that includesthe high priority DICOM priority header field can be sent back to themedical picture archive system 2620 to replace or otherwise be mapped tothe original DICOM image of the head CT scan.

In various embodiments, the medical picture archive system 2620 isdisconnected from network 2630, for example, to comply with requirementsregarding Protected Health Information (PHI), such as patientidentifiers and other private patient information included in the DICOMimages and/or otherwise stored by the medical picture archive system2620. The medical picture archive integration system 2600 can enableprocessing of DICOM images while still protecting private patientinformation by first de-identifying DICOM data by utilizingde-identification system 2608. The de-identification system 2608 canutilize designated processors and memory of the medical picture archiveintegration system, for example, designated for PHI. Thede-identification system 2608 can be decoupled from the network 2630 toprevent the DICOM images that still include patient identifiers frombeing accessed via the network 2630. For example, as shown in FIG. 8A,the de-identification system 2608 is not connected to network interface2606. Furthermore, only the de-identification system 2608 has access tothe original DICOM files received from the medical picture archivesystem 2620 via receiver 2602. The de-identified image storage system2610 and annotating system 2612, as they are connected to network 2630via network interface 2606, only store and have access to thede-identified medical scan produced by the de-identification system2608.

This containerization that separates the de-identification system 2608from the de-identified image storage system 2610 and the annotatingsystem 2612 is further illustrated in FIG. 8B, which presents anembodiment of the medical picture archive integration system 2600. Thede-identification system 2608 can include its own designated memory 2654and processing system 2652, connected to receiver 2602 via bus 2659. Forexample, this memory 2654 and processing system 2652 can be designatedfor PHI, and can adhere to requirements for handling PHI. The memory2654 can store executable instructions that, when executed by theprocessing system 2652, enable the de-identification system to performthe de-identification function on DICOM images received via receiver2602 of the de-identification system. The incoming DICOM images can betemporarily stored in memory 2654 for processing, and patientidentifiers detected in performing the de-identification function can betemporarily stored in memory 2654 to undergo anonymization. Interface2655 can transmit the de-identified medical scan to interface 2661 foruse by the de-identified image storage system 2610 and the annotatingsystem 2612. Interface 2655 can be protected from transmitting originalDICOM files and can be designated for transmission of de-identifiedmedical scan only.

Bus 2669 connects interface 2661, as well as transmitter 2604 andnetwork interface 2606, to the de-identified image storage system 2610and the annotating system 2612. The de-identified image storage system2610 and annotating system 2612 can utilize separate processors andmemory, or can utilize shared processors and/or memory. For example, thede-identified image storage system 2610 can serve as temporary memory ofthe annotating system 2612 as de-identified images are received andprocessed to generate annotation data.

As depicted in FIG. 8B, the de-identified image storage system 2610 caninclude memory 2674 that can temporarily store incoming de-identifiedmedical scans as it undergoes processing by the annotating system 2612and/or can archive a plurality of de-identified medical scanscorresponding to a plurality of DICOM images received by the medicalpicture archive integration system 2600. The annotating system 2612 caninclude a memory 2684 that stores executable instructions that, whenexecuted by processing system 2682, cause the annotating system 2612perform a first inference function on de-identified medical scan togenerate annotation data by utilizing the model parameters received viainterface 2606, and to generate an annotated DICOM file based on theannotation data for transmission via transmitter 2604. The modelparameters can be stored in memory 2684, and can include modelparameters for a plurality of inference functions, for example,corresponding to a set of different scan categories.

The medical picture archive integration system can be an onsite system,installed at a first geographic site, such as a hospital or othermedical entity that is affiliated with the medical picture archivesystem 2620. The hospital or other medical entity can further beresponsible for the PHI of the de-identification system, for example,where the memory 2654 and processing system 2652 are owned by,maintained by, and/or otherwise affiliated with the hospital or othermedical entity. The central server system 2640 can be located at asecond, separate geographic site that is not affiliated with thehospital or other medical entity and/or at a separate geographic sitethat is not affiliated with the medical picture archive system 2620. Thecentral server system 2640 can be a server configured to be outside thenetwork firewall and/or out outside the physical security of thehospital or other medical entity or otherwise not covered by theparticular administrative, physical and technical safeguards of thehospital or other medical entity.

FIG. 8C further illustrates how model parameters can be updated overtime to improve existing inference functions and/or to add new inferencefunctions, for example corresponding to new scan categories. Inparticular, the some or all of the de-identified medical scans generatedby the de-identification system 2608 can be transmitted back to thecentral server system, and the central server system 2640 can train onthis data to improve existing models by producing updated modelparameters of an existing inference function and/or to generate newmodels, for example, corresponding to new scan categories, by producingnew model parameters for new inference functions. For example, thecentral server system 2640 can produce updated and/or new modelparameters by performing the training step 1352 of the medical scanimage analysis system 112, as discussed in conjunction with FIG. 7A, ona plurality of de-identified medical scans received from the medicalpicture archive integration system 2600.

The image type parameters can be determined by the central server systemto dictate characteristics of the set of de-identified medical scans tobe received to train and/or retrain the model. For example, the imagetype parameters can correspond to one or more scan categories, canindicate scan classifier data 420, can indicate one or more scanmodalities, one or more anatomical regions, a date range, and/or otherparameters. The image type parameters can be determined by the centralserver system based on training parameters 620 determined for thecorresponding inference function to be trained, and/or based oncharacteristics of a new and/or existing scan category corresponding tothe inference function to be trained. The image type parameters can besent to the medical picture archive integration system 2600, and arequest such as a DICOM query can be sent to the medical picture archivesystem 2620, via transmitter 2604, that indicates the image typeparameters. For example, the processing system 2682 can be utilized togenerate the DICOM query based on the image type parameters receivedfrom the central server system 2640. The medical picture archive systemcan automatically transmit one or more DICOM images to the medicalpicture archive integration system in response to determining that theone or more DICOM images compares favorably to the image typeparameters. The DICOM images received in response can be de-identifiedby the de-identification system 2608. In some embodiments, thede-identified medical scans can be transmitted directly to the centralserver system 2640, for example, without generating annotation data.

The central server system can generate the new and/or updated modelparameters by training on the received set of de-identified medicalscans, and can transmit the new and/or updated model parameters to thede-identified storage system. If the model parameters correspond to anew inference function for a new scan category, the medical picturearchive integration system 2600 can generate a request, such as a DICOMquery, for transmission to the medical picture archive system indicatingthat incoming scans corresponding to image type parameters correspondingto the new scan category be sent to the medical picture archiveintegration system. The annotating system can update the set ofinference functions to include the new inference function, and theannotating system can select the new inference function from the set ofinference functions for subsequently generated de-identified medicalscans by the de-identification system by determining each of thesede-identified medical scans indicate the corresponding DICOM imagecorresponds to the new scan category. The new model parameters can beutilized to perform the new inference function on each of thesede-identified medical scans to generate corresponding annotation data,and an annotated DICOM file corresponding to each of these de-identifiedmedical scans can be generated for transmission to the medical picturearchive system via the transmitter.

In some embodiments, the central server system 2640 receives a pluralityof de-identified medical scans from a plurality of medical picturearchive integration system 2600, for example, each installed at aplurality of different hospitals or other medical entities, via thenetwork 2630. The central server system can generate training sets byintegrating de-identified medical scans from some or all of theplurality of medical picture archive integration systems 2600 to trainone or more inference functions and generate model parameters. Theplurality of medical picture archive integration systems 2600 canutilize the same set of inference functions or different sets ofinference functions. In some embodiments, the set of inference functionsutilized by the each of the plurality of medical picture archive systems2620 are trained on different sets of training data. For example, thedifferent sets of training data can correspond to the set ofde-identified medical scans received from the corresponding medicalpicture archive integration system 2600.

In some embodiments, the medical scan diagnosing system 108 can beutilized to implement the annotating system 2612, where thecorresponding subsystem processing device 235 and subsystem memorydevice 245 of the medical scan diagnosing system 108 are utilized toimplement the processing system 2682 and the memory 2684, respectively.Rather than receiving the medical scans via the network 150 as discussedin conjunction with FIG. 6A, the medical scan diagnosing system 108 canperform a selected medical scan inference function 1105 on an incomingde-identified medical scan generated by the de-identification system2608 and/or retrieved from the de-identified image storage system 2610.Memory 2684 can store the set of medical scan inference functions 1105,each corresponding to a scan category 1120, where the inference functionis selected from the set based on determining the scan category of thede-identified medical scan and selecting the corresponding inferencefunction. The processing system 2682 can perform the selected inferencefunction 1105 to generate the inference data 1110, which can be furtherutilized by the annotating system 2612 to generate the annotated DICOMfile for transmission back to the medical picture archive system 2620.New medical scan inference functions 1105 can be added to the set whencorresponding model parameters are received from the central serversystem. The remediation step 1140 can be performed locally by theannotating system 2612 and/or can be performed by the central serversystem 2640 by utilizing one or more de-identified medical scans andcorresponding annotation data sent to the central server system 2640.Updated model parameters can be generated by the central server system2640 and sent to the medical picture archive integration system 2600 asa result of performing the remediation step 1140.

The central server system 2640 can be implemented by utilizing one ormore of the medical scan subsystems 101, such as the medical scan imageanalysis system 112 and/or the medical scan diagnosing system 108, toproduce model parameters for one or more inference functions. Thecentral server system can store or otherwise communicate with a medicalscan database 342 that includes the de-identified medical scans and/orannotation data received from one or more medical picture archiveintegration systems 2600. Some or all entries of the medical scandatabase 342 can be utilized to as training data to produce modelparameters for one or more inference functions. These entries of themedical scan database 342 can be utilized by other subsystems 101 asdiscussed herein. For example, other subsystems 101 can utilize thecentral server system 2640 to fetch medical scans and/or correspondingannotation data that meet specified criteria. The central server system2640 can query the medical picture archive integration system 2600 basedon this criteria, and can receive de-identified medical scans and/orannotation data in response. This can be sent to the requestingsubsystem 101 directly and/or can be added to the medical scan database342 or another database of the database storage system 140 for access bythe requesting subsystem 101.

Alternatively or in addition, the central server system 2640 can storeor otherwise communicate with a user database 344 storing user profileentries corresponding to each of a plurality of medical entities thateach utilize a corresponding one of a plurality of medical picturearchive integration systems 2600. For example, basic user datacorresponding to the medical entity can be stored as basic user data, anumber of scans or other consumption information indicating usage of oneor more inference functions by corresponding medical picture archiveintegration system can be stored as consumption usage data, and/or anumber of scans or other contribution information indicatingde-identified scans sent to the central server system as training datacan be stored as contribution usage data. The user profile entry canalso include inference function data, for example, with a list of modelparameters or function identifiers, such as medical scan analysisfunction identifiers 357, of inference functions currently utilized bythe corresponding medical picture archive integration system 2600. Theseentries of the user database 344 can be utilized by other subsystems 101as discussed herein.

Alternatively or in addition, the central server system 2640 can storeor otherwise communicate with a medical scan analysis function database346 to store model parameters, training data, or other information forone or more inference functions as medical scan analysis functionentries 356. In some embodiments, model parameter data 623 can indicatethe model parameters and function classifier data 610 can indicate thescan category of inference function entries. In some embodiments, themedical scan analysis function entry 356 can further include usageidentifying information indicating a medical picture archive integrationsystem identifier, medical entity identifier, and/or otherwiseindicating which medical archive integration systems and/or medicalentities have received the corresponding model parameters to utilize theinference function corresponding to the medical scan analysis functionentry 356. These entries of the medical scan analysis function database346 can be utilized by other subsystems 101 as discussed herein.

In some embodiments, the de-identification function is a medical scananalysis function, for example, with a corresponding medical scananalysis function entry 356 in the medical scan analysis functiondatabase 346. In some embodiments, the de-identification function istrained by the central server system 2640. For example, the centralserver system 2640 can send de-identification function parameters to themedical picture archive integration system 2600 for use by thede-identification system 2608. In embodiments with a plurality ofmedical picture archive integration systems 2600, each of the pluralityof medical picture archive integration systems 2600 can utilize the sameor different de-identification functions. In some embodiments, thede-identification function utilized by the each of the plurality ofmedical picture archive integration systems 2600 are trained ondifferent sets of training data. For example, the different sets oftraining data can correspond to each different set of de-identifiedmedical scans received from each corresponding medical picture archiveintegration system 2600.

In some embodiments, as illustrated in FIGS. 8D-8F, the medical picturearchive integration system 2600 can further communicate with a reportdatabase 2625, such as a Radiology Information System (RIS), thatincludes a plurality of medical reports corresponding to the DICOMimages stored by the medical picture archive system 2620.

As shown in FIG. 8D, the medical picture archive integration system 2600can further include a receiver 2603 that receives report data,corresponding to the DICOM image, from report database 2625. The reportdatabase 2625 can be affiliated with the medical picture archive system2620 and can store report data corresponding to DICOM images stored inthe medical picture archive system. The report data of report database2625 can include PHI, and the report database 2625 can thus bedisconnected from network 2630.

The report data can include natural language text, for example,generated by a radiologist that reviewed the corresponding DICOM image.The report data can be used to generate the de-identified medical scan,for example, where the de-identification system 2608 performs a naturallanguage analysis function on the report data to identify patientidentifying text in the report data. The de-identification system 2608can utilize this patient identifying text to detect matching patientidentifiers in the DICOM image to identify the patient identifiers ofthe DICOM image and generate the de-identified medical scan. In someembodiments, the report data can be de-identified by obfuscating,hashing, removing, replacing with a fiducial, or otherwise anonymizingthe identified patient identifying text to generate de-identified reportdata.

The de-identified report data can be utilized by the annotating system2612, for example, in conjunction with the DICOM image, to generate theannotation data. For example, the annotating system 2612 can perform anatural language analysis function on the de-identified natural languagetext of the report data to generate some or all of the annotation data.In some embodiments, the de-identified report data is sent to thecentral server system, for example, to be used as training data forinference functions, for natural language analysis functions, for othermedical scan analysis functions, and/or for use by at least one othersubsystem 101. For example, other subsystems 101 can utilize the centralserver system 2640 to fetch medical reports that correspond toparticular medical scans or otherwise meet specified criteria. Thecentral server system 2640 can query the medical picture archiveintegration system 2600 based on this criteria, and can receivede-identified medical reports in response. This can be sent to therequesting subsystem 101 directly, can be added to the medical scandatabase 342, a de-identified report database, or another database ofthe database storage system 140 for access by the requesting subsystem101.

In some embodiments the medical picture archive integration system 2600can query the report database 2625 for the report data corresponding toa received DICOM image by utilizing a common identifier extracted fromthe DICOM image.

In some embodiments, the report data can correspond to a plurality ofDICOM images. For example, the report data can include natural languagetext describing a plurality of medical scans of a patient that caninclude multiple sequences, multiple modalities, and/or multiple medicalscans taken over time. In such embodiments, the patient identifying textand/or annotation data detected in the report data can also be appliedto de-identify and/or generate annotation data for the plurality ofDICOM images it describes. In such embodiments, the medical picturearchive integration system 2600 can query the medical picture archivesystem 2620 for one or more additional DICOM images corresponding to thereport data, and de-identified data and annotation data for theseadditional DICOM images can be generated accordingly by utilizing thereport data.

In some embodiments, as shown in FIG. 8E, the medical picture archivesystem 2620 communicates with the report database 2625. The medicalpicture archive system 2620 can request the report data corresponding tothe DICOM image from the report database 2625, and can transmit thereport data to the medical picture archive integration system 2600 via aDICOM communication protocol for receipt via receiver 2602. The medicalpicture archive system 2620 can query the report database 2625 for thereport data, utilizing a common identifier extracted from thecorresponding DICOM image, in response to determining to send thecorresponding DICOM image to the medical picture archive integrationsystem 2600.

FIG. 8F presents an embodiment where report data is generated by theannotating system 2612 and is transmitted, via a transmitter 2605, tothe report database 2625, for example via a DICOM communication protocolor other protocol recognized by the report database 2625. In otherembodiments, the report data is instead transmitted via transmitter 2604to the medical picture archive system 2620, and the medical picturearchive system 2620 transmits the report data to the report database2625.

The report data can be generated by the annotating system 2612 as outputof performing the inference function on the de-identified medical scan.The report data can include natural language text data 448 generatedautomatically based on other diagnosis data 440 such as abnormalityannotation data 442 determined by performing the inference function, forexample, by utilizing a medical scan natural language generatingfunction trained by the medical scan natural language analysis system114. The report data can be generated instead of, or in addition to, theannotated DICOM file.

FIG. 9 presents a flowchart illustrating a method for execution by amedical picture archive integration system 2600 that includes a firstmemory and a second memory that store executional instructions that,when executed by at least one first processor and at least one secondprocessor, respectfully, cause the medical picture archive integrationsystem to perform the steps below. In various embodiments, the firstmemory and at least one first processor are implemented by utilizing,respectfully, the memory 2654 and processing system 2652 of FIG. 8B. Invarious embodiments, the second memory is implemented by utilizing thememory 2674 and/or the memory 2684 of FIG. 8B. In various embodiments,the at least one second processor is implemented by utilizing theprocessing system 2682 of FIG. 8B.

Step 2702 includes receiving, from a medical picture archive system viaa receiver, a first DICOM image for storage in the first memory,designated for PHI, where the first DICOM image includes at least onepatient identifier. Step 2704 includes performing, via at least onefirst processor coupled to the first memory and designated for PHI, ade-identification function on the first DICOM image to identify the atleast one patient identifier and generate a first de-identified medicalscan that does not include the at least one patient identifier.

Step 2706 includes storing the first de-identified medical scan in asecond memory that is separate from the first memory. Step 2708 includesreceiving, via a network interface communicating with a network thatdoes not include the medical picture archive system, first modelparameters from a central server.

Step 2710 includes retrieving the first de-identified medical scan fromthe second memory. Step 2712 includes utilizing the first modelparameters to perform a first inference function on the firstde-identified medical scan to generate first annotation data via atleast one second processor that is different from the at least one firstprocessor. Step 2714 includes generating, via the at least one secondprocessor, a first annotated DICOM file for transmission to the medicalpicture archive system via a transmitter, where the first annotatedDICOM file includes the first annotation data and further includes anidentifier that indicates the first DICOM image. In various embodiments,the first annotated DICOM file is a DICOM presentation state file.

In various embodiments, the second memory further includes operationalinstructions that, when executed by the at least one second processor,further cause the medical picture archive integration system to retrievea second de-identified medical scan from the de-identified image storagesystem, where the second de-identified medical scan was generated by theat least one first processor by performing the de-identificationfunction on a second DICOM image received from the medical picturearchive system. The updated model parameters are utilized to perform thefirst inference function on the second de-identified medical scan togenerate second annotation data. A second annotated DICOM file isgenerated for transmission to the medical picture archive system via thetransmitter, where the second annotated DICOM file includes the secondannotation data and further includes an identifier that indicates thesecond DICOM image.

In various embodiments, the second memory stores a plurality ofde-identified medical scans generated by the at least one firstprocessor by performing the de-identification function on acorresponding plurality of DICOM images received from the medicalpicture archive system via the receiver. The plurality of de-identifiedmedical scans is transmitted to the central server via the networkinterface, and the central server generates the first model parametersby performing a training function on training data that includes theplurality of de-identified medical scans.

In various embodiments, the central server generates the first modelparameters by performing a training function on training data thatincludes a plurality of de-identified medical scans received from aplurality of medical picture archive integration systems via thenetwork. Each of the plurality of medical picture archive integrationsystems communicates bidirectionally with a corresponding one of aplurality of medical picture archive systems, and the plurality ofde-identified medical scans corresponds to a plurality of DICOM imagesstored by the plurality of medical picture archive integration systems.

In various embodiments, the first de-identified medical scan indicates ascan category of the first DICOM image. The second memory further storesoperational instructions that, when executed by the at least one secondprocessor, further cause the medical picture archive integration systemto select the first inference function from a set of inference functionsbased on the scan category. The set of inference functions correspondsto a set of unique scan categories that includes the scan category. Invarious embodiments, each unique scan category of the set of unique scancategories is characterized by one of a plurality of modalities and oneof a plurality of anatomical regions.

In various embodiments, the first memory further stores operationalinstructions that, when executed by the at least one first processor,further cause the medical picture archive integration system to receivea plurality of DICOM image data from the medical picture archive systemvia the receiver for storage in the first memory in response to a querytransmitted to the medical picture archive system via the transmitter.The query is generated by the medical picture archive integration systemin response to a request indicating a new scan category received fromthe central server via the network. The new scan category is notincluded in the set of unique scan categories, and the plurality ofDICOM image data corresponds to the new scan category. Thede-identification function is performed on the plurality of DICOM imagedata to generate a plurality of de-identified medical scans fortransmission to the central server via the network.

The second memory further stores operational instructions that, whenexecuted by the at least one second processor, further cause the medicalpicture archive integration system to receive second model parametersfrom the central server via the network for a new inference functioncorresponding to the new scan category. The set of inference functionsis updated to include the new inference function. The secondde-identified medical scan is retrieved from the first memory, where thesecond de-identified medical scan was generated by the at least onefirst processor by performing the de-identification function on a secondDICOM image received from the medical picture archive system. The newinference function is selected from the set of inference functions bydetermining the second de-identified medical scan indicates the secondDICOM image corresponds to the new scan category. The second modelparameters are utilized to perform the new inference function on thesecond de-identified medical scan to generate second annotation data. Asecond annotated DICOM file is generated for transmission to the medicalpicture archive system via the transmitter, where the second annotatedDICOM file includes the second annotation data and further includes anidentifier that indicates the second DICOM image.

In various embodiments, the medical picture archive integration systemgenerates parameter data for transmission to the medical picture archivesystem that indicates the set of unique scan categories. The medicalpicture archive system automatically transmits the first DICOM image tothe medical picture archive integration system in response todetermining that the first DICOM image compares favorably to one of theset of unique scan categories.

In various embodiments, the second memory further stores operationalinstructions that, when executed by the at least one second processor,cause the medical picture archive integration system to generate anatural language report data is based on the first annotation data andto transmit, via a second transmitter, the natural language report datato a report database associated with the medical picture archiveintegration system, where the natural language report data includes anidentifier corresponding to the first DICOM image.

In various embodiments, the first memory further stores operationalinstructions that, when executed by the at least one first processor,cause the medical picture archive integration system to receive, via asecond receiver, a natural language report corresponding to the firstDICOM image from the report database. A set of patient identifying textincluded in the natural language report are identified. Performing thede-identification function on the first DICOM image includes searchingthe first DICOM image for the set of patient identifying text toidentify the at least one patient identifier.

In various embodiments, the first memory is managed by a medical entityassociated with the medical picture archive system. The medical picturearchive integration system is located at a first geographic sitecorresponding to the medical entity, and the central server is locatedat a second geographic site. In various embodiments, the first memory isdecoupled from the network to prevent the first DICOM image thatincludes the at least one patient identifier from being communicated viathe network. In various embodiments, the medical picture archive systemis a Picture Archive and Communication System (PACS) server, and thefirst DICOM image is received in response to a query sent to the medicalpicture archive system by the transmitter in accordance with a DICOMcommunication protocol.

FIG. 10A presents an embodiment of a de-identification system 2800. Thede-identification system 2800 can be utilized to implement thede-identification system 2608 of FIGS. 8A-8F. In some embodiments, thede-identification system 2800 can be utilized by other subsystems tode-identify image data, medical report data, private fields of medicalscan entries 352 such as patient identifier data 431, and/or otherprivate fields stored in databases of the database memory device 340.

The de-identification system can be operable to receive, from at leastone first entity, a medical scan and a medical report corresponding tothe medical scan. A set of patient identifiers can be identified in asubset of fields of a header of the medical scan. A header anonymizationfunction can be performed on each of the set of patient identifiers togenerate a corresponding set of anonymized fields. A de-identifiedmedical scan can be generated by replacing the subset of fields of theheader of the medical scan with the corresponding set of anonymizedfields.

A subset of patient identifiers of the set of patient identifiers can beidentified in the medical report by searching text of the medical reportfor the set of patient identifiers. A text anonymization function can beperformed on the subset of patient identifiers to generate correspondinganonymized placeholder text for each of the subset of patientidentifiers. A de-identified medical report can be generated byreplacing each of the subset of patient identifiers with thecorresponding anonymized placeholder text. The de-identified medicalscan and the de-identified medical report can be transmitted to a secondentity via a network.

As shown in FIG. 10A, the de-identification system 2800 can include atleast one receiver 2802 operable to receive medical scans, such asmedical scans in a DICOM image format. The at least one receiver 2802 isfurther operable to receive medical reports, such as report data 449 orother reports containing natural language text diagnosing, describing,or otherwise associated the medical scans received by thede-identification system. The medical scans and report data can bereceived from the same or different entity, and can be received by thesame or different receiver 2802 in accordance with the same or differentcommunication protocol. For example, the medical scans can be receivedfrom the medical picture archive system 2620 of FIGS. 8A-8F and thereport data can be received from the report database 2625 of FIGS.8D-8F. In such embodiments, the receiver 2802 can be utilized toimplement the receiver 2602 of FIG. 8B.

The de-identification system 2800 can further include a processingsystem 2804 that includes at least one processor, and a memory 2806. Thememory 2806 can store operational instructions that, when executed bythe processing system, cause the de-identification system to perform atleast one patient identifier detection function on the received medicalscan and/or the medical report to identify a set of patient identifiersin the medical scan and/or the medical report. The operationalinstructions, when executed by the processing system, can further causethe de-identification system to perform an anonymization function on themedical scan and/or the medical report to generate a de-identifiedmedical scan and/or a de-identified medical report that do not includethe set of patient identifiers found in performing the at least onepatient identifier detection function. Generating the de-identifiedmedical scan can include generating a de-identified header andgenerating de-identified image data, where the de-identified medicalscan includes both the de-identified header and the de-identified imagedata. The memory 2806 can be isolated from Internet connectivity, andcan be designated for PHI.

The de-identification system 2800 can further include at least onetransmitter 2808, operable to transmit the de-identified medical scanand de-identified medical report. The de-identified medical scan andde-identified medical report can be transmitted back to the same entityfrom which they were received, respectively, and/or can be transmittedto a separate entity. For example, the at least one transmitter cantransmit the de-identified medical scan to the de-identified imagestorage system 2610 of FIGS. 8A-8F and/or can transmit the de-identifiedmedical scan to central server system 2640 via network 2630 of FIGS.8A-8F. In such embodiments, the transmitter 2808 can be utilized toimplement the interface 2655 of FIG. 8B. The receiver 2802, processingsystem 2804, memory 2806, and/or transmitter 2808 can be connected viabus 2810.

Some or all of the at least one patient identifier detection functionand/or at least one anonymization function as discussed herein can betrained and/or implemented by one or subsystems 101 in the same fashionas other medical scan analysis functions discussed herein, can be storedin medical scan analysis function database 346 of FIG. 3 , and/or canotherwise be characterized by some or all fields of a medical scananalysis function entry 356 of FIG. 5 .

The de-identification system 2800 can perform separate patientidentifier detection functions on the header of a medical report and/ormedical scan, on the text data of the medical report, and/or on theimage data of the medical scan, such as text extracted from the imagedata of the medical scan. Performance of each of these functionsgenerates an output of its own set of identified patient identifiers.Combining these sets of patient identifiers yields a blacklist term set.A second pass of the header of a medical report and/or medical scan, onthe text data of the medical report, and/or on the image data of themedical scan that utilizes this blacklist term set can catch any termsthat were missed by the respective patient identifier detectionfunction, and thus, the outputs of these multiple identificationprocesses can support each other. For example, some of the data in theheaders will be in a structured form and can thus be easier to reliablyidentify. This can be exploited and used to further anonymize theseidentifiers when they appear in free text header fields, report data,and/or in the image data of the medical scan. Meanwhile, unstructuredtext in free text header fields, report data, and/or image data of themedical scan likely includes pertinent clinical information to bepreserved in the anonymization process, for example, so it can beleveraged by at least one subsystems 101 and/or so it can be leveragedin training at least one medical scan analysis function.

At least one first patient identifier detection function can includeextracting the data in a subset of fields of a DICOM header, or anotherheader or other metadata of the medical scan and/or medical report witha known type that corresponds to patient identifying data. For example,this patient identifying subset of fields can include a name field, apatient ID number field or other unique patient identifier field, a datefield, a time field, an age field, an accession number field, SOPinstance UID, and/or other fields that could be utilized to identify thepatient and/or contain private information. A non-identifying subset offields of the header can include hospital identifiers, machine modelidentifiers, and/or some or all fields of medical scan entry 352 that donot correspond to patient identifying data. The patient identifyingsubset of fields and the non-identifying subset of fields can bemutually exclusive and collectively exhaustive with respect to theheader. The at least one patient identifier function can includegenerating a first set of patient identifiers by ignoring thenon-identifying subset of fields and extracting the entries of thepatient identifying subset of fields only. This first set of patientidentifiers can be anonymized to generate a de-identified header asdiscussed herein.

In some embodiments, at least one second patient identifier detectionfunction can be performed on the report data of the medical report. Theat least one second patient identifier detection function can includeidentifying patient identifying text in the report data by performing anatural language analysis function, for example, trained by the medicalscan natural language analysis system 114. For example, the at least onesecond patient identifier detection function can leverage the knownstructure of the medical report and/or context of the medical report. Asecond set of patient identifiers corresponding to the patientidentifying text can be determined, and the second set of patientidentifiers can be anonymized to generate a de-identified medicalreport. In some embodiments, a de-identified medical report includesclinical information, for example, because the portion of the originalmedical report that includes the clinical information was deemed to befree of patient identifying text and/or because the portion of theoriginal medical report that includes the clinical information wasdetermined to include pertinent information to be preserved.

In some embodiments, the medical report includes image datacorresponding to freehand or typed text. For example the medical reportcan correspond to a digitized scan of original freehand text written bya radiologist or other medical professional. In such embodiments, thepatient identifier detection function can first extract the text fromthe freehand text in the image data to generate text data before the atleast one second patient identifier detection function is performed onthe text of the medical report to generate the second set of patientidentifiers.

In some embodiments, the at least one second patient identifierdetection function can similarly be utilized to identify patientidentifying text in free text fields and/or unstructured text fields ofa DICOM header and/or other metadata of the medical scan and/or medicalreport data by performing a natural language analysis function, forexample, trained by the medical scan natural language analysis system114. A third set of patient identifiers corresponding to this patientidentifying text of the free text and/or unstructured header fields canbe determined, and the third set of patient identifiers can beanonymized to generate de-identified free text header field and/orunstructured header fields. In some embodiments, a de-identified freetext header field and/or unstructured header field includes clinicalinformation, for example, because the portion of the originalcorresponding header field that includes the clinical information wasdeemed to be free of patient identifying text and/or because the portionof the original corresponding header field that includes the clinicalinformation was determined to include pertinent information to bepreserved.

Patient identifiers can also be included in the image data of themedical scan itself. For example, freehand text corresponding to apatient name written on a hard copy of the medical scan beforedigitizing can be included in the image data, as discussed inconjunction with FIG. 10B. Other patient identifiers, such asinformation included on a patient wristband or other identifyinginformation located on or within the vicinity of the patient may havebeen captured when the medical scan was taken, and can thus be includedin the image. At least one third patient identifier detection functioncan include extracting text from the image data and/or detectingnon-text identifiers in the image data by performing a medical scanimage analysis function, for example, trained by the medical scan imageanalysis system 112. For example, detected text that corresponds to animage location known to include patient identifiers, detected text thatcorresponds to a format of a patient identifier, and/or or detected textor other image data determined to correspond to a patient identifier canbe identified. The at least one third patient identifier detectionfunction can further include identifying patient identifying text in thetext extracted from the image data by performing the at least one secondpatient identifier detection function and/or by performing a naturallanguage analysis function. A fourth set of patient identifierscorresponding to patient identifying text or other patient identifiersdetected in the image data of the medical scan can be determined, andthe fourth set of patient identifiers can be anonymized in the imagedata to generate de-identified image data of the medical scan asdescribed herein. In particular, the fourth set of patient identifierscan be detected in a set of regions of image data of the medical scan,and the set of regions of the image data can be anonymized.

In some embodiments, only a subset of the patient identifier detectionfunctions described herein are performed to generate respective sets ofpatient identifiers for anonymization. In some embodiments, additionalpatient identifier detection functions can be performed on the medicalscan and/or medical report to determine additional respective sets ofpatient identifiers for anonymization. The sets of patient identifiersoutputted by performing each patient identifier detection function canhave a null or non-null intersection. The sets of patient identifiersoutputted by performing each patient identifier function can have nullor non-null set differences.

Cases where the sets of patient identifiers have non-null setdifferences can indicate that a patient identifier detected by onefunction may have been missed by another function. The combined set ofpatient identifiers, for example, generated as the union of the sets ofsets of patient identifiers outputted by performing each patientidentifier function, can be used to build a blacklist term set, forexample, stored in memory 2806. The blacklist term set can designate thefinal set of terms to be anonymized. A second pass of header data,medical scans, medical reports, and/or any free text extracted from theheader data, the medical scan, and/or the medical report can beperformed by utilizing the blacklist term set to flag terms foranonymization that were not caught in performing the respective at leastone patient identifier detection function. For example, performing thesecond pass can include identifying at least one patient identifier ofthe blacklist term set in the header, medical report, and/or image dataof the medical scan. This can include by searching correspondingextracted text of the header, medical report, and/or image data forterms included in blacklist term set and/or by determining if each termin the extracted text is included in the blacklist term set.

In some embodiments, at least one patient identifier is not detecteduntil the second pass is performed. Consider an example where a freetext field of a DICOM header included a patient name that was notdetected in performing a respective patient identifier detectionfunction on the free text field of the DICOM header. However, thepatient name was successfully identified in the text of the medicalreport in performing a patient identifier detection function on themedical report. This patient name is added to the blacklist term list,and is detected in a second pass of the free text field of the DICOMheader. In response to detection in the second pass, the patient name ofthe free text field of the DICOM header can be anonymized accordingly togenerate a de-identified free text field. Consider a further examplewhere the patient name is included in the image data of the medicalscan, but was not detected in performing a respective patient identifierdetection function on the free text field of the DICOM header. In thesecond pass, this patient name can be detected in at least one region ofimage data of the medical scan by searching the image data for theblacklist term set.

In some embodiments, performing some or all of the patient identifierdetection functions includes identifying a set of non-identifying terms,such as the non-identifying subset of fields of the header. Inparticular, the non-identifying terms can include terms identified asclinical information and/or other terms determined to be preserved. Thecombined set of non-identifying terms, for example, generated as theunion of the sets of sets of non-identifying outputted by performingeach patient identifier function, can be used to build a whitelist termset, for example, stored in memory 2806. Performing the second pass canfurther include identifying at least one non-identifying term of thewhitelist term set in the header, medical report, and/or image data ofthe medical scan, and determining not to anonymize, or to otherwiseignore, the non-identifying term.

In various embodiments, some or all terms of the whitelist term set canbe removed from the blacklist term set. In particular, at least one termpreviously identified as a patient identifier in performing one or morepatient identifier detection functions is determined to be ignored andnot anonymized in response to determining the term is included in thewhitelist term set. This can help ensure that clinically importantinformation is not anonymized, and is thus preserved in thede-identified medical scan and de-identified medical report.

In some embodiments, the second pass can be performed after each of thepatient identifier detection functions are performed. For example,performing the anonymization function can include performing this secondpass by utilizing the blacklist term set to determine the final set ofterms to be anonymized. New portions of text in header fields, notpreviously detected in generating the first set of patient identifiersor the third set of patient identifiers, can be flagged foranonymization by determining these new portions of text correspond toterms of the blacklist term set. New portions of text the medicalreport, not previously detected in generating in the second set ofpatient identifiers, can be flagged for anonymization by determiningthese new portions of text correspond to terms of the blacklist termset. New regions of the image data of the medical scan, not previouslydetected in generating the fourth set of patient identifiers, can beflagged for anonymization by determining these new portions of textcorrespond to terms of the blacklist term set.

In some embodiments, the blacklist term set is built as each patientidentifier detection function is performed, and performance ofsubsequent patient identifier detection functions includes utilizing thecurrent blacklist term set. For example, performing the second patientidentifier detection function can include identifying a first subset ofthe blacklist term set in the medical report by searching the text ofthe medical report for the blacklist term set and/or by determining ifeach term in the text of the medical report is included in the blacklistterm set. Performing the second patient identifier detection functioncan further include identifying at least one term in the medical reportthat is included in the whitelist term set, and determining to ignorethe term in response. The first subset can be anonymized to generate thede-identified medical report as discussed herein. New patientidentifiers not already found can be appended to the blacklist term set,and the updated blacklist term set can be applied to perform a secondsearch of the header and/or image data of the medical scan, and at leastone of the new patient identifiers can be identified in the header inthe second search of the header and/or in the image data in a secondsearch of the image data. These newly identified patient identifiers inthe header and/or image data are anonymized in generating thede-identified medical scan.

As another example, a second subset of the blacklist term set can bedetected in a set of regions of image data of the medical scan byperforming the medical scan image analysis function on image data of themedical scan, where the image analysis function includes searching theimage data for the set of patient identifiers. For example, the medicalscan image analysis function can include searching the image data fortext, and the second subset can include detected text that matches oneor more terms of the blacklist term set. In some embodiments, detectedtext that matches one or more terms of the whitelist term set can beignored. The second subset can be anonymized to generate de-identifiedimage data as discussed herein. New patient identifiers that aredetected can be appended to the blacklist term set, and the updatedblacklist term set can be applied to perform a second search of theheader and/or metadata of the medical scan, and/or can be applied toperform a second search of the medical report. At least one of the newpatient identifiers can be identified in the header as a result ofperforming the second search of the header and/or at least one of thenew patient identifiers can be identified medical report as a result ofperforming the second search of the medical report. These newlyidentified patient identifiers can be anonymized in the header alongwith the originally identified blacklist term set in generating thede-identified header, and/or can be anonymized in the medical reportalong with the originally identified first subset in generating thede-identified medical report.

In some embodiments, the memory 2806 further stores a global blacklist,for example, that includes a vast set of known patient identifyingterms. In some embodiments, the global blacklist is also utilized by atleast one patient identifier detection function and/or in performing thesecond pass to determine patient identifying terms for anonymization. Insome embodiments, the blacklist term set generated for a particularmedical scan and corresponding medical report can be appended to theglobal blacklist for use in performing the second pass and/or indetecting patient identifiers in subsequently received medical scansand/or medical reports.

Alternatively or in addition, the memory 2806 can further store a globalwhitelist, for example, that includes a vast set of terms that can beignored. In particular, the global whitelist can include clinical termsand/or other terms that are deemed beneficial to preserve that do notcorrespond to patient identifying information. In some embodiments, theglobal whitelist is utilized by at least one patient identifierdetection function and/or in performing the second pass to determineterms to ignore in the header, image data, and/or medical report. Insome embodiments, the whitelist term set generated for a particularmedical scan and corresponding medical report can be appended to theglobal whitelist for use in performing the second pass and/or inignoring terms in subsequently received medical scans and/or medicalreports.

Alternatively or in addition, the memory 2806 can further store a globalgraylist, for example, that includes ambiguous terms that could bepatient identifying terms in some contexts, but non-identifying terms inother contexts. For example, “Parkinson” could correspond to patientidentifying data if part of a patient name such as “John Parkinson”, butcould correspond to non-patient identifying data meant to be ignored andpreserved in the de-identified medical report and/or de-identifiedmedical scan if part of a diagnosis term such as “Parkinson's disease.”In some embodiments, the global graylist is also utilized in performingthe second pass and/or in performing at least one patient identifierdetection function to determine that a term is included in the graylist,and to further determine whether the term should be added to theblacklist term set for anonymization or whitelist term set to be ignoredby leveraging context of accompanying text, by leveraging known datatypes of a header field from which the term was extracted, by leveragingknown structure of the term, by leveraging known data types of alocation of the image data from which the term was extracted, and/or byleveraging other contextual information. In some embodiments, thegraylist term set can be updated based on blacklist and/or whitelistterm sets for a particular medical scan and corresponding medicalreport.

In some embodiments, the at least one anonymization function includes afiducial replacement function. For example, some or all of the blacklistterm set can be replaced with a corresponding, global fiducial in theheader, report data, and/or image data. In some embodiments, the globalfiducial can be selected from a set of global fiducials based on a typeof the corresponding patient identifier. Each patient identifierdetected in the header and/or medical report can be replaced with acorresponding one of the set of global text fiducials. Each patientidentifiers detected in the image data can be replaced with acorresponding one of the set of global image fiducials. For example, oneor more global image fiducials can overlay pixels of regions of theimage data that include the identifying patient data, to obfuscate theidentifying patient data in the de-identified image data.

The global text fiducials and/or global image fiducials can berecognizable by inference functions and/or training functions, forexample, where the global text fiducials and global image fiducials areignored when processed in a training step to train an inference functionand/or are ignored in an inference step when processed by an inferencefunction. Furthermore, the global text fiducials and/or global imagefiducials can be recognizable by a human viewing the header, medicalreport, and/or image data. For example, a radiologist or other medicalprofessional, upon viewing a header, medical report, and/or image data,can clearly identify the location of a patient identifier that wasreplaced by the fiducial and/or can identify the type of patientidentifier that was replaced by the fiducial.

As an example, the name “John Smith” can be replaced in a header and/ormedical report with the text “% PATIENT NAME %”, where the text “%PATIENT NAME %” is a global fiducial for name types of the header and/orthe text of medical reports. The training step and/or inference step ofmedical scan natural language analysis functions can recognize andignore text that matches “% PATIENT NAME %” automatically.

FIG. 10B illustrates an example of anonymizing patient identifiers inimage data of a medical scan. In this example, the name “John Smith” andthe date “May 4, 2010” is detected as freehand text in the originalimage data of a medical scan. The regions of the image data that includethe patient identifiers can each be replaced by global fiducial in theshape of a rectangular bar, or any other shape. As shown in FIG. 10B, afirst region corresponding to the location of “John Smith” in theoriginal image data is replaced by fiducial 2820 in the de-identifiedimage data, and a second region corresponding to the location of “May 4,2010” in the original image data is replaced by fiducial 2822 in thede-identified image data. The size, shape, and/or location of eachglobal visual fiducial can be automatically determined based on thesize, shape, and/or location of the region that includes the patientidentifier to minimize the amount of the image data that is obfuscated,while still ensuring the entirety of the text is covered. While notdepicted in FIG. 10B, the fiducial can be of a particular color, forexample, where pixels of the particular color are automaticallyrecognized by the training step and/or inference step of medical scanimage analysis functions to indicate that the corresponding region beignored, and/or where the particular color is not included in theoriginal medical scan and/or is known to not be included in any medicalscans. The fiducial can include text recognizable to human inspectionsuch as “% PATIENT NAME” and “% DATE” as depicted in FIG. 10B, and/orcan include a QR code, logo, or other unique symbol recognizable tohuman inspection and/or automatically recognizable by the training stepand/or inference step of medical scan image analysis functions toindicate that the corresponding region be ignored.

In some embodiments, other anonymization functions can be performed ondifferent ones of the patient identifying subset of fields to generatethe de-identified header, de-identified report data, and/orde-identified image data. For example, based on the type of identifyingdata of each field of the header, different types of headeranonymization functions and/or text anonymization functions can beselected and utilized on the header fields, text of the report, and/ortext extracted from the image data. A set of anonymization functions caninclude a shift function, for example, utilized to offset a date, timeor other temporal data by a determined amount to preserve absolute timedifference and/or to preserve relative order over multiple medical scansand/or medical reports of a single patient. FIG. 10B depicts an examplewhere the shift function is performed on the date detected in the imagedata to generate fiducial 2822, where the determined amount is 10 yearsand 1 month. The determined amount can be determined by thede-identification system randomly and/or pseudo-randomly for eachpatient and/or for each medical scan and corresponding medical report,ensuring the original date cannot be recovered by utilizing a knownoffset. In various embodiments, other medical scans and/or medicalreports are fetched for the same patient by utilizing a patient IDnumber or other unique patient identifier of the header. These medialscans and reports can be anonymized as well, where the dates and/ortimes detected in these medical scans and/or medical reports offset bythe same determined amount, randomized or pseudo-randomized forparticular patient ID number, for example, based on performing a hashfunction on the patient ID number.

The set of anonymization functions can include at least one hashfunction, for example utilized to hash a unique patient ID such as apatient ID number, accession number, and/or SOP instance UID of theheader and/or text. In some embodiments, the hashed SOP instance UID,accession number, and/or patient ID number are prepended with a uniqueidentifier, stored in a database of the memory 2806 and/or shared withthe entities to which the de-identified medical scans and/or medicalreports are transmitted, so that de-identified medical scans and theircorresponding de-identified medical reports can be linked and retrievedretroactively. Similarly, longitudinal data can be preserved as multiplemedical scans and/or medical reports of the same patient will beassigned the same hashed patient ID.

The set of anonymization functions can further include at least onemanipulator function for some types of patient identifiers. Some valuesof header fields and/or report text that would normally not beconsidered private information can be considered identifying patientdata if they correspond to an outlier value or other rare value thatcould then be utilized to identify the corresponding patient from a verysmall subset of possible options. For example, a patient age over 89could be utilized to determine the identity of the patient, for example,if there are very few patients over the age of 89. To prevent suchcases, in response to determining that a patient identifier correspondsto an outlier value and/or in response to determining that a patientidentifier compares unfavorably to a normal-range threshold value, thepatient identifier can be capped at the normal-range threshold value orcan otherwise be manipulated. For example, a normal-range thresholdvalue corresponding to age can be set at 89, and generating ade-identified patient age can include capping patient ages that arehigher than 89 at 89 and/or can include keeping the same value forpatient ages that are less than or equal to 89.

In some embodiments, the de-identified header data is utilized toreplace the corresponding first subset of patient identifiers detectedin the medical report with text of the de-identified header fields. Inother embodiments, a set of text anonymization functions includes aglobal text fiducial replacement function, shift function, a hashfunction, and/or manipulator functions that anonymize the correspondingtypes of patient identifiers in the medical report separately.

In some embodiments where the image data of a medical scan includes ananatomical region corresponding to a patient's head, the image data mayinclude an identifying facial structure and/or facial features thatcould be utilized to determine the patient's identity. For example, adatabase of facial images, mapped to a corresponding plurality of peopleincluding the patient, could be searched and a facial recognitionfunction could be utilized to identify the patient in the database.Thus, facial structure included in the image data can be consideredpatient identifying data.

To prevent this problem and maintain patient privacy, thede-identification system can further be implemented to perform facialobfuscation for facial structure detected in medical scans. At least oneregion of the image data that includes identifying facial structure canbe determined by utilizing a medical image analysis function. Forexample, the medical image analysis function can include a facialdetection function that determines the regions of the image data thatinclude identifying facial structure based on searching the image datafor pixels with a density value that corresponds to facial skin, facialbone structure, or other density of an anatomical mass type thatcorresponds to identifying facial structure, and the facial obfuscationfunction can be performed on the identified pixels. Alternatively or inaddition, the facial detection function can determine the region basedon identifying at least one shape in the image data that corresponds toa facial structure.

The image obfuscation function can include a facial structureobfuscation function performed on the medical scan to generatede-identified image data that does not include identifying facialstructure. For example, the facial structure obfuscation function canmask, scramble, replace with a fiducial, or otherwise obfuscate thepixels of the region identified by the facial detection function. Insome embodiments, the facial structure obfuscation function can performa one-way function on the region that preserves abnormalities of thecorresponding portions of the image, such as nose fractures or facialskin legions, while still obfuscating the identifying facial structuresuch that the patient is not identifiable. For example, the pixels ofthe identifying facial structure can be altered such that they convergetowards a fixed, generic facial structure. In some embodiments, aplurality of facial structure image data of a plurality of patients canbe utilized to generate the generic facial structure, for example,corresponding to an average or other combination of the plurality offaces. For example, the pixels of the generic facial structure can beaveraged with, superimposed upon, or otherwise combined with the pixelsof the region of the image data identified by the facial detectionfunction in generating the de-identified image data.

In some embodiments, a hash function can be performed on an average ofthe generic facial structure and the identified facial structure of theimage data so that the generic facial structure cannot be utilized inconjunction with the resulting data of the de-identified image data toreproduce the original, identifying facial structure. In suchembodiments, the hash function can alter the pixel values while stillpreserving abnormalities. In some embodiments, a plurality of random,generic facial structures can be generated by utilizing the plurality offacial structure image data, for example, where each if the plurality offacial structure image data are assigned a random or pseudo-randomweight in an averaging function utilized to create the generic facialstructure, where a new, random or pseudo-random set of weights aregenerated each time the facial structure obfuscation function isutilized to create a new, generic facial structure to be averaged withthe identified facial structure in creating the de-identified image datato ensure the original identifying facial structure cannot be extractedfrom the resulting de-identified image data.

While facial obfuscation is described herein, similar techniques can beapplied in a similar fashion to other anatomical regions that aredetermined to include patient identifiers and/or to other anatomicalregions that can be utilized to extract patient identifying informationif not anonymized.

In some embodiments, the at least one receiver 2802 is included in atleast one transceiver, for example, enabling bidirectional communicationbetween the medical picture archive system 2620 and/or the reportdatabase 2625. In such embodiments, the de-identification system 2800can generate queries to the medical picture archive system 2620 and/orthe report database 2625 for particular medical scans and/or medicalreports, respectively. In particular, if the medical scan and medicalreport are stored and/or managed by separate memories and/or separateentities, they may not be received at the same time. However, a linkingidentifier, such as DICOM identifiers in headers or metadata of themedical scan and/or medical report, such accession number, patient IDnumber, SOP instance UID, or other linking identifier that maps themedical scan to the medical report can be utilized to fetch a medicalreport corresponding to a received medical scan and/or to fetch amedical scan corresponding to a received medical report via a query sentutilizing the at least one transceiver. For example, in response toreceiving the medical scan from the medical picture archive system 2620,the de-identification system can extract a linking identifier from aDICOM header of the medical scan, and can query the report database 2625for the corresponding medical report by indicating the linkingidentifier in the query. Conversely, in response to receiving themedical report from the report database 2625, the de-identificationsystem can extract the linking identifier from a header, metadata,and/or text body of the medical report, and can query the medicalpicture archive system 2620 for the corresponding medical scan byindicating the linking identifier in the query. In some embodiments, amapping of de-identified medical scans to original medical scans, and/ora mapping of de-identified medical reports to original medical reportscan be stored in memory 2806. In some embodiments, linking identifierssuch as patient ID numbers can be utilized to fetch additional medicalscans, additional medical reports, or other longitudinal datacorresponding to the same patient.

FIG. 11 presents a flowchart illustrating a method for execution by ade-identification system 2800 that stores executional instructions that,when executed by at least one processor, cause the de-identification toperform the steps below.

Step 2902 includes receiving from a first entity, via a receiver, afirst medical scan and a medical report corresponding to the medicalscan. Step 2904 includes identifying a set of patient identifiers in asubset of fields of a first header of the first medical scan. Step 2906includes performing a header anonymization function on each of the setof patient identifiers to generate a corresponding set of anonymizedfields. Step 2908 includes generating a first de-identified medical scanby replacing the subset of fields of the first header of the firstmedical scan with the corresponding set of anonymized fields. Step 2910includes identifying a first subset of patient identifiers of the set ofpatient identifiers in the medical report by searching text of themedical report for the set of patient identifiers. Step 2912 includesperforming a text anonymization function on the first subset of patientidentifiers to generate corresponding anonymized placeholder text foreach of the first subset of patient identifiers. Step 2914 includesgenerating a de-identified medical report by replacing each of the firstsubset of patient identifiers with the corresponding anonymizedplaceholder text. Step 2916 includes transmitting, via a transmitter,the de-identified first medical scan and the de-identified medicalreport to a second entity via a network.

In various embodiments, the medical scan is received from a PictureArchive and Communication System (PACS), where the medical report isreceived from a Radiology Information System (RIS), and where the firstde-identified medical scan and the de-identified medical report aretransmitted to a central server that is not affiliated with the PACS orthe RIS. In various embodiments, first medical scan and the medicalreport are stored in a first memory for processing. The first memory isdecoupled from the network to prevent the set of patient identifiersfrom being communicated via the network. The first de-identified medicalscan and the de-identified medical report are stored in a second memorythat is separate from the first memory. The first de-identified medicalscan and the de-identified medical report are fetched from the secondmemory for transmission to the second entity.

In various embodiments, the header anonymization function performed oneach of the set of patient identifiers is selected from a plurality ofheader anonymization functions based on one of a plurality of identifiertypes of the corresponding one of the subset of fields. In variousembodiments, the plurality of identifier types includes a date type. Ashift function corresponding to the date type is performed on a firstdate of the first header to generate the first de-identified medicalscan, where the shift function includes offsetting the first date by adetermined amount. A second medical scan is received, via the receiver,that includes a second header. A unique patient ID of the first headermatches a unique patient ID of the second header. The shift function isperformed on a second date of the second header by offsetting the seconddate by the determined amount to generate a second de-identified medicalscan. The second de-identified medical scan is transmitted to the secondentity via the network.

In various embodiments, the plurality of identifier types includes aunique patient ID type. A hash function corresponding the unique patientID type is performed on the unique patient ID of the first header togenerate the first de-identified medical scan. The hash function isperformed on the unique patient ID of the second header to generate thesecond de-identified medical scan. An anonymized unique patient ID fieldof the first de-identified medical scan matches an anonymized uniquepatient ID field of the second de-identified medical scan as a result ofthe unique patient ID of the first header matching the unique patient IDof the second header.

In various embodiments, the plurality of identifier types includes alinking identifier type that maps the medical scan to the medicalreport. A hash function corresponding to the linking identifier type isperformed on a linking identifier of the first header to generate ahashed linking identifier. A linking identifier field of the firstde-identified medical scan includes the hashed linking identifier.Performing the text anonymization function on the first subset ofpatient identifiers includes determining one of the first subset ofpatient identifiers corresponds to linking identifier text andperforming the hash function on the one of the first subset of patientidentifiers to generate the hashed linking identifier, where thede-identified medical report includes the hashed linking identifier.

In various embodiments, a second subset of patient identifiers of theset of patient identifiers is identified in a set of regions of imagedata of the medical scan by performing an image analysis function onimage data of the medical scan. The image analysis function includessearching the image data for the set of patient identifiers. Anidentifier type is determined for each of the second subset of patientidentifiers. One of a plurality of image fiducials is selected for eachof the second subset of patient identifiers based on the identifiertype. De-identified image data is generated, where a set of regions ofthe de-identified image data, corresponding to the set of regions of theimage data, includes the one of the plurality of image fiducials toobfuscate each of the second subset of patient identifiers. Generatingthe first de-identified medical scan further includes replacing theimage data of the medical scan with the de-identified image data.

In various embodiments, a new patient identifier is identified in themedical report by performing a natural language analysis function on themedical report, where new patient identifier is not included in the setof patient identifiers. The set of patient identifiers is updated toinclude the new patient identifier prior to searching the image data ofthe medical scan for the set of patient identifiers, and the secondsubset of patient identifiers includes the new patient identifier.

In various embodiments, the memory further stores a global identifierblacklist. The natural language analysis function includes searching themedical report for a plurality of terms included in the globalidentifier blacklist to identify the new patient identifier. In variousembodiments, the de-identification system determines that the globalidentifier blacklist does not include one of the set of patientidentifiers, and the global identifier blacklist is updated to includethe one of the set of patient identifiers.

In various embodiments, performing the image analysis function furtherincludes identifying a new patient identifier in the image data, wherenew patient identifier is not included in the set of patientidentifiers. Identifying text is extracted from a region of the imagedata corresponding to the new patient identifier. The new patientidentifier is identified in the medical report by searching text of themedical report for the identifying text. The text anonymization functionis performed on new patient identifier to generate anonymizedplaceholder text for the new patient identifier. Generating thede-identified medical report further includes replacing the identifyingtext with the anonymized placeholder text for the new patientidentifier.

In various embodiments, generating the de-identified image data furtherincludes detecting an identifying facial structure in the image data ofthe medical scan. Generating the de-identified image data includesperforming a facial structure obfuscation function on the image data,and where the de-identified image data does not include the identifyingfacial structure.

FIG. 12A illustrates an embodiment of an intensity transformaugmentation system 3002. The intensity transform augmentation system3002 can be utilized to perform density windowing, adjust the brightnessor contrast, or otherwise perform at least one intensity transformationfunction on medical scans of a training set prior to training a computervision model. A medical scan training set can be received by theintensity transform augmentation system 3002, for example, from medicalscan database 342. Transformation function parameter data is alsoreceived and/or determined locally. A pre-processing step 3005 utilizesthe transformation function parameter data to perform one or moreintensity transformation functions, indicated by the transformationfunction parameter data, on some or all medical scans in the trainingset to produce at least one augmented image as output of at least oneintensity transformation function for some or all of the medical scansin the training set. The augmented images can replace the originalmedical scans to produce the pre-processed training set, and/or cansupplement the original medical scans to produce the pre-processedtraining set. A training step 3010 can be performed on the pre-processedtraining set to generate a computer vision model. Model parameters orother data associated with the model can be stored locally for furtheruse, can be sent directly to one or more other subsystems 101 for use,can be sent to a client device 120 for display, and/or can betransmitted for storage in a corresponding medical scan analysisfunction entry in the medical scan analysis function database 346.

Some or all of the steps and/or processes performed by the intensitytransform augmentation system 3002 as discussed herein can be performedautomatically, without human intervention. Alternatively or in addition,some or all of the step and/or processed performed by the intensitytransform augmentation system 3002 as discussed herein can be performedin response to user input received from a client device, correspondingto user interaction with the client device in response to a promptdisplayed by an interface of a display device. Such prompts can begenerated by the intensity transform augmentation system 3002 fortransmission to the client device.

As shown in FIG. 12A, the intensity transform augmentation system 3002can communicate bi-directionally, via network 150, with the medical scandatabase 342, medical scan analysis function database 346, and/or otherdatabases of the database storage system 140, with one or more clientdevices 120, and/or, while not shown in FIG. 12A, one or more subsystems101 of FIG. 1 . In some embodiments, the intensity transformaugmentation system 3002 is an additional subsystem 101 of the medicalscan processing system 100, implemented by utilizing the subsystemmemory device 245, subsystem processing device 235, and/or subsystemnetwork interface 265 of FIG. 2A. In some embodiments, the intensitytransform augmentation system 3002 is implemented by utilizing, orotherwise communicates with, the central server 2640. For example, themodel data of one or more computer vision models generated by theintensity transform augmentation system 3002 can be utilized as one ormore inference functions sent to the annotating system of the medicalpicture archive integration system 2600. As another example, some or allof the databases of the database storage system 140 are populated withde-identified data generated by the medical picture archive integrationsystem 2600. In some embodiments, the intensity transform augmentationsystem 3002 can receive de-identified medical scans, annotation data,and/or reports directly from the medical picture archive integrationsystem 2600. For example, the intensity transform augmentation system3002 can request de-identified medical scans, annotation data, and/orreports that match requested criteria, for example, corresponding totraining set criteria. In some embodiments, some or all of the intensitytransform augmentation system 3002 is implemented by utilizing othersubsystems 101 and/or is operable to perform functions or otheroperations described in conjunction with one or more other subsystems101. In some embodiments, the intensity transform augmentation system3002 is integrated within and/or utilizes the medical scan imageanalysis system 112.

The transformation function parameter data is depicted as beingretrieved from medical scan analysis function database 346, for example,where some or all intensity transformation functions are stored asmedical scan analysis function entries 356 and/or are trained and/orutilized as medical scan analysis functions as described herein. In someembodiments, the transformation function parameter data is stored and/orupdated locally. In some embodiments, some or all of the transformationfunction parameter data is determined based on to user input to a clientdevice 120, for example, in response to a prompt displayed by a displaydevice of the client device, and sent to the intensity transformaugmentation system 3002. For example, a user can generate and/or modifyone or more intensity transformation functions and/or can select one ormore intensity transforms to be utilized from a pre-determined intensitytransformation function set. The transformation function parameter datacan indicate one or more intensity transformation functions to beapplied to the medical scan, as discussed in further detail in FIGS.12B-12F.

The training step 3010 can correspond to some or all steps of thetraining step 1352 of FIG. 7A. The training step 3010 can be performedby utilizing the medical scan image analysis system 112. The trainingstep 3010 can include training a neural network or other machinelearning model. Output label data can be received in conjunction withthe medical scan training set, and can be utilized to train the model.For example, the output label data can be received in conjunction withthe medical scan training set, for example, corresponding to annotationdata, diagnosis data, report data, metadata of the medical scan, and/orfields of medical scan entries of the medical scans in the training set.The same output label data originally assigned to each medical scan inthe training set can be assigned to each corresponding augmented imageproduced in the pre-processing step for utilization in training step3010. In some embodiments, the output label is utilized to select thesubset of the set of intensity transformation functions that will beapplied and/or to otherwise determine one or more of the intensitytransformation functions that will be utilized, for example, based ondensity properties of an abnormality indicated in the output labeland/or a region of interest indicated in the output label.

The training set can correspond to the training set data 621 of acorresponding medical scan function entry for the model to be generated.Medical scans to be included in the training set can be determined bythe intensity transform augmentation system 3002 based on correspondingcriteria for the model to be trained and/or can be indicated via userinput to a client device via a user interface displayed on a displaydevice.

The model data can correspond to a computer vision model discussed inconjunction with the medical scan image analysis system 112, discussedwith one or more other subsystems as discussed herein, and/or anycomputer vision model that is trained on medical scan image data and cangenerate abnormality data or other inference data for new medical scans.In some embodiments, the computer vision model is a neural network.

FIG. 12B illustrates an example of performing pre-processing step 3005.In some embodiments, a single augmented image is generated for eachmedical scan in the training set by utilizing an intensitytransformation function, and the same or different intensitytransformation function can be performed on each medical scan. In otherembodiments, a plurality of augmented images can be generated for eachmedical scan in the training set by utilizing a set of intensitytransformation functions 1-n, where the set of intensity transformationfunctions are the same or different for each medical scan. In someembodiments, the set of intensity transformation functions includes anidentity function, and/or the un-processed image data of the medicalscan itself can otherwise be included in the pre-processed training set.

In some embodiments, some or all of the set of intensity transformationfunctions correspond to one of a plurality of different anatomyfeatures, for example, present in the training set of medical scans. Inparticular, each intensity transformation function can be based on thedensity properties of the corresponding anatomy feature, for example, toaccentuate the corresponding anatomy feature adjusting a contrast and/orbrightness of image data of the one of the training set of medicalscans. The different anatomy features can include, for example, air,fat, soft tissue, bone, blood, other fluids, parenchyma, gallstone,and/or foreign body features. Some of the different anatomy features cancorrespond to abnormalities included in the training set of medicalscans. The different anatomy features can correspond to mutuallyexclusive density value ranges.

The transformation function parameter data can indicate the set ofintensity transformation functions, and/or different transform functionparameter data can indicate each of the set of intensity transformationfunctions. The transformation function parameter data can furtherindicate input conditions for some or all of the set of intensitytransformation functions, for example, where a subset of the set ofintensity transformation functions are selected from the set to beperformed on the medical scan. The input conditions can correspond toscan type or other criteria corresponding to each medical scan, and/orcan correspond to randomly generated conditions. For example, thepre-processing step 3005 can include determining a subset of the set ofintensity transformation functions to be performed on each medical scanof the training set based on the modality of the medical scan, theanatomical region of the medical scan, the patient history or otherpatient data corresponding to the medical scan, longitudinal datareceived along with the medical scan, metadata of the medical scan,density properties of an a anatomical feature included in the medicalscan, density properties of an abnormality indicated in diagnosis dataor other metadata of the medial scan, density properties of ananatomical feature identified as being of interest, and/or a densityvalue histogram of the medical scan. The scan classifier data 420,diagnosis data 440, and/or other fields of the medical scan entry can beutilized to dictate which subset of functions are utilized.

For example, a first subset of the intensity transformation functionscan be determined for a first medical scan of the training set, and asecond subset of intensity transformation functions can be determinedfor a second medical scan of the training set, where the set differencebetween the first subset and the second subset are non-null. The firstand second subset can be selected based on different anatomy featurespresent in different anatomical regions of the first medical scan andsecond medical scans, respectively. Alternatively or in addition, thefirst and second subset can be selected based on density properties ofdifferent abnormalities indicated in the output label of the first andsecond medical scans, respectively. Alternatively or in addition, thefirst and second subset can be selected based on different anatomyfeatures corresponding to different regions of interest indicated inoutput label of the first and second medical scans, respectively. Asanother example, the pre-processing step 3005 can include determiningsome or all of the transformation function parameter data randomlyand/or pseudo-randomly in accordance with a uniform distribution orother pre-determined random distribution.

In some embodiments, the input to some or all intensity transformationfunctions includes a plurality of pixel values of the image data of themedical scan, and the output is a plurality of corresponding pixelvalues of an augmented image. For example, a plurality of pixels of theimage data of the medical scan can each indicate a pixel value, whichcan correspond to a density value such as a Hounsfield unit value and/orraw sensor data captured in generating the medical scan. In someembodiments, the plurality of pixels of the image data of the medicalscan instead correspond to greyscale values, RGB values, or other valuesin accordance with an image format such as a JPEG format. For example,contrasting parameters and/or density windowing may have already beenapplied and/or the image data may have been undergone otherpre-processing to convert density values to greyscale values. In someembodiments, the input is determined based on the display parameter data470, for example, based on density window data 475.

In some embodiments, a data format of the intensity transformationfunction output is different from a data format of the intensitytransformation function input. For example, the input can correspond todensity values of raw sensor data, and the output can correspond togreyscale values of a JPEG. In other embodiments, the data format of theintensity transformation function output is the same as the image formatof the intensity transformation function input. For example, the inputcan correspond to greyscale values of a JPEG, and the output cancorrespond to greyscale values of a new JPEG. In such embodiments, thenew JPEG can be generated by utilizing the intensity transformationfunction to alter contrast and/or brightness of the input JPEG, tootherwise re-contrast the JPEG, to apply an image filter to the inputJPEG, and/or otherwise alter the pixel values of the input JPEG. Some orall of the set of intensity transformation functions can utilize thesame or different type of input format. Some or all of the set ofintensity transformation functions can generate the same or differenttype of output format. In some embodiments, every one of the set ofintensity transformation functions produces the same output image formatthat corresponds to an input image format of the training step 3010and/or an input image format to the final model generated by theintensity transform augmentation system 3002.

A plurality of pixel values for each of a plurality of pixels of anaugmented image can be generated by performing the intensitytransformation function on each of the pixel values of each of theplurality of corresponding pixels of the medical scan. Each pixel valuecan be processed independently of the neighboring pixels, where theoutput pixel value is a deterministic mapping of input pixel value. Thedeterministic mapping of the intensity transformation function can be inaccordance with a linear function, a piecewise function, a non-linearfunction, and/or any deterministic function. The deterministic mappingof the intensity transformation function can be a monotonicallyincreasing function. In some embodiments, the intensity transformationfunction is reversible, and the input medical scan can be regenerated byperforming an inverse transformation function. In other embodiments,some input pixel values, such as one or more similar pixel values, aremapped to the same output pixel value, and/or the original input cannotbe regenerated.

In some embodiments, some or all medical scans of the training setcorrespond to medical scans that include a plurality of image slices412. In such embodiments, each of the set of intensity transformationfunctions can be performed on all of the plurality of image slices. Insuch embodiments, each augmented image can include a plurality ofaugmented image slices. The plurality of augmented image slices producedby a same one of the intensity transformation functions can be groupedtogether in the same set of augmented image slices to be fed into themodel as training data, or can be processed separately.

In some embodiments, the training set includes a plurality of medicalstudies, and the model is trained on the plurality of medical studies.Each of the plurality of medical studies can include longitudinal data,such as longitudinal data 433, that includes set of medical scans. Thesame and/or different set of intensity transformation functions can beperformed on each medical scan in the longitudinal data of a medicalstudy to produce corresponding augmented longitudinal data that includesa set of augmented images for each of the medical scans in thelongitudinal data. Each of the plurality of augmented images produced bya same one of the intensity transformation functions can be groupedtogether in the same set of augmented longitudinal data to be fed intothe model as training data, or can be processed separately.

FIG. 12C illustrates an example intensity transformation function 3050that utilizes a density windowing function. A density window boundarypair 3052 can designate which subrange of densities within a total rangeof possible density values are assigned a pixel value in the image data.The density window width 3054 and density window center 3056, aredefined by a given density window boundary pair 3052, and similarly, thedensity window boundary pair 3052 can be determined by a given densitywindow width 3054 and the density window center 3056. Thus, the densitywindow boundary pair 3052, density window width 3054, and/or a densitywindow center 3056 can be utilized as intensity transformation functionparameters. In some embodiments, as shown in FIG. 12C, the deterministicmapping is in accordance with a linear function with a an input domainthat spans the entirety of density values bounded by the density windowboundary pair 3052 and an output range that spans the entirety ofpossible pixel values and/or is otherwise bounded by a maximum andminimum pixel value. In some embodiments, the density window center 3056indicates the density value that is mapped to a midpoint value of thespan of output pixel values. In some embodiments, density values thatare less than the lower bound of the density window boundary pair aremapped to the minimum pixel value, and/or density values that aregreater than the upper bound of the density window boundary pair aremapped to the maximum pixel value. In some embodiments, the densitywindowing corresponds to density windowing techniques discussed inconjunction with the medical scan image analysis system 112.

In some embodiments, the density window for a particular intensitytransformation function can be based on the density properties of acorresponding anatomy feature, for example, to accentuate thecorresponding anatomy feature. In some embodiments, a known averagedensity value of the anatomy feature is utilized to determine thedensity window center, for example, where the density window center isset to the known average density value. In some embodiments, a knownstandard deviation of density of the anatomy feature is utilized todetermine the value density window width, where the width is set to alarger value for higher standard deviations. In some embodiments, adensity value of an abnormality of an output label for a particularmedical scan in the training set and/or average density value of anabnormality classification corresponding to the known abnormality in themedical scan is utilized to determine the density window.

FIGS. 12D-12F illustrate additional examples of intensity transformationfunctions 3050 that can be utilized in accordance with variousembodiments. FIG. 12D illustrates an example where the deterministicmapping is still defined in accordance with a density window boundarypair, but also utilizes a non-linear deterministic mapping within thedensity window. For example, the non-linear deterministic mapping can beindicated by a mean and a standard deviation included in thetransformation function parameter data. Similar to the density windowcenter, the mean can correspond to a input density value that is mappedto the midpoint output pixel value. The standard deviation can indicatehow the density values will be distributed across the span of outputpixel values in the mapping. Rather than a linear mapping within adensity window boundary pair 3052, a deterministic mapping can be inaccordance with a cumulative distribution function (CDF) of the normalfunction as depicted in FIG. 12D, where the density window center isassigned the mean value, and where the density window boundary pair 3052is assigned to three standard deviations higher and lower than thedensity window center (or alternatively, the density window width isassigned to six times the standard deviation). As another example wherethe intensity transformation function is in accordance with a normaldistribution, density values that fall within bounds a first standarddeviation lower and higher than the mean can be mapped to approximately68 percent of the output pixel values, centered at the midpoint outputpixel value, density values that fall within bounds a second deviationlower and higher than the mean can be mapped to approximately 95 percentof the output pixel values, centered at the midpoint output pixel value,and/or density values that fall within bounds a third standard deviationlower and higher than the mean can be mapped to approximately 99.7percent of the output pixel values. The non-linear function can be inaccordance with a CDF of another distribution that can correspond to asymmetric or skewed distribution, and can be defined by a mean, standarddeviation, and/or other parameters of the other distribution indicatedin the parameter data.

FIG. 12E illustrates an example where the deterministic mapping of anintensity transformation function 3050 is a non-linear function thatspans the entire range of possible density values of the input medicalscan, for example, where the entire range of possible density values areutilized and no windowing boundaries are applied. FIG. 12F illustratesan example where the deterministic mapping of an intensitytransformation function 3050 is a non-linear function that maps otherpixel values, such as greyscale color values of an input image, to pixelvalues of the output. For example, both the input and output image dataof the intensity transformation function 3050 can have a same imageformat. Some or all of the set of intensity transformation functions 1-ncan utilize examples illustrated in FIGS. 12C-12F, and/or can utilizeother function types. In some embodiments, every one of the set ofintensity transformation functions is a different, deterministic,monotonically increasing mapping of input pixel value to output pixelvalue. The transformation function parameter data can indicate a CDF,formula, function coefficients, and/or can otherwise indicate and/ordefine the deterministic mapping of each of the set of intensitytransformation functions.

In some embodiments, the transformation function parameter data canindicate desired distribution data of the output pixel values, and thedesired distribution data is utilized to determine the deterministicmapping, given the image data of the input medical scan. In particular,the desired distribution data can indicate a shape, distributionparameters such as a distribution function and/or mean and standarddeviation, and/or boundary data indicating the desired output histogramof pixel values. While an exact match of the desired distribution datamay be unreasonable, defined or predetermined bounds indicating anallowable deviation from the desired distribution can be indicated, andthe final distribution of the output histogram can be in accordance withthe bounds. In such embodiments, a histogram of the density valuesand/or other value type of the input pixels is utilized to determine adistribution of the input pixel values. By utilizing this distributionof input pixel values and the desired output distribution, the intensitytransformation function 3050 that, when applied to input pixel values ofthe determined input distribution, generates this desired outputdistribution. In this fashion, the intensity transformation function canbe determined separately for each medical scan, based on their own pixeldistribution data. In some embodiments, the desired output histogram isin accordance with a uniform distribution. In other embodiments, thedesired output histogram is in accordance with a normal distribution orany other distribution, which can be indicated by a mean, standarddeviation, and/or other parameters in the transformation functionparameter data.

In some embodiments, such intensity transformation functions 3050 thatutilize desired distribution data of the output pixels can bepre-determined, for example, based on pre-determined grouping of scanswith similar pixel value histograms. In some embodiments, thesepre-determined groupings are grouped by scan type, such as modality,anatomical region, and/or other scan classifier data 420. For example,the pre-determined groupings can be formed in response to determiningthese scan type groupings have similar density value distributions. Insome embodiments, other custom groupings are determined, for example, byutilizing a clustering algorithm. Intensity transformation functions canbe generated for an average input pixel distribution of each of thesepre-determined groups, where the intensity transformation functiongenerates the desired output distribution for this average input pixeldistribution. Some or all of the set of intensity transformationfunctions 1-n can be generated in this fashion for some or all medicalscans in the training set and/or for medical scans processed inaccordance with FIG. 12G. Medical scans can be assigned to a mostsimilar one of the groups, for example, by performing a similarityfunction and/or by utilizing the scan classifier data 420. Thepre-determined deterministic mapping for the assigned group can beutilized on the scan in performing the corresponding intensitytransformation function.

In some embodiments, some or all of the transformation functionparameter data for some or all of the set of intensity transformationfunction 1-n are determined via user input to client device 120 viainteraction with a user interface displayed by a display device. In someembodiments, some or all of the transformation function parameter datafor some or all of the set of intensity transformation function 1-n aredetermined as part of the transform parameter evaluation step discussedin conjunction with FIG. 12H.

In some embodiments, one or more intensity transformation functions aredetermined randomly. For example, some or all of the intensitytransformation function parameters that dictate the deterministicmapping can be generated via a random or pseudo-random process. Inparticular, for each medical scan in the training set, some or all ofthe intensity transformation function parameters can be determinedrandomly, separately, for one or more intensity transformation functionsto be performed on each of the medical scans in the training set. Forexample, the density window boundary pair (or alternatively, the densitywindow center and density window width) of a density window of theintensity transformation function can be determined via a random orpseudo-random selection. As another example, the mean and standarddeviation dictating a normal CDF or other CDF of the intensitytransformation function can be determined via a random or pseudo-randomselection. As another example, a single one, or proper subset of sizegreater than one, of the set of pre-defined intensity transformationfunctions 1-n as discussed in FIG. 12B can be selected from the set viaa random or pseudo-random selection. As another example, parametersdictating the shape and/or other features of the desired outputhistogram utilized to generate the intensity transformation function canbe determined via a random or pseudo-random selection.

In some embodiments, the random or pseudo-random selection is inaccordance with a pre-defined random distribution, which can be the sameor different for each of the set of intensity transformation functionparameters. The pre-defined random distribution can be uniform, normal,or can be in accordance with another random distribution. For example,for one of the intensity transformation functions that corresponds to aparticular anatomical feature, pre-defined mean values for the randomselection of some or all parameters can be set based on the densityproperties of the particular anatomical feature and/or the values ofparameters for the corresponding one of the set of intensitytransformation functions 1-n. Similarly, known density ranges and/orknown density variability of the particular anatomical feature can beutilized to determine a pre-defined standard deviation for the randomselection of some or all parameters. The random selection of eachparameter can correspond to a uniform, normal, or other distributioncentered at the pre-defined mean value and/or pre-defined standarddeviation value of the corresponding parameter. In some embodiments, therandom intensity transformation function parameters are givenpre-defined means corresponding to given intensity transformationfunction parameters and/or learned intensity transform functionparameters as discussed in accordance with the FIG. 12H, and normal orother random distribution utilized in random selection of each parameterin accordance with the pre-defined means.

In some embodiments, the random distribution for one or more parametersand/or for selection of one or more intensity transformation functionsfrom the set is determined separately for each medical scan, forexample, based on the scan type, anatomical region, and/or other scanclassifier data 420 or metadata associated with the medical scan, andeach of the parameters are randomly selected in accordance with eachdetermined corresponding random distribution.

In some embodiments, some or all of the intensity transformationfunction parameters for random selection, and/or some or all of therandom distributions dictating random selection of the intensitytransformation function parameters, are determined via user input toclient device 120 via interaction with a user interface displayed by adisplay device. In some embodiments, some or all of the intensitytransformation function parameters for random selection, and/or some orall of the random distributions dictating random selection of theintensity transformation function parameters, are determined as part ofthe transform parameter evaluation step discussed in conjunction withFIG. 12H. For example, optimal random distributions for selection ofeach of the intensity transformation function parameters and/or forselection of one of the set of intensity transformation function can bedetermined, and the random intensity transformation function parametersutilized for some or all of the training set of medical scans aregenerated in accordance with the optimal random distribution.

FIG. 12G illustrates an embodiment of an intensity transformaugmentation system 3002 that is operable to generate inference data fornew medical scans once the model has been generated as discussed inconjunction with FIG. 12A. In this fashion, new medical scans canundergo the same and/or similar pre-processing as the medical scans ofthe training set utilized to generate the model before being fed intothe model themselves. In some embodiment, a different subsystem 101 isresponsible for generating inference data for new medical scans.

The intensity transform augmentation system 3002 can receive a newmedical scan, for example, not included in the training set, forprocessing. The medical scan can be retrieved from the same or differentmedical scan database 342 as the training set, can be received from aclient device 120 and/or another subsystem 101, and/or can be otherwisereceived for processing. A pre-processing step 3006 can be applied tothe incoming medical scan. The pre-processing step 3006 can includeutilizing transformation function parameter data to produce apre-processed medical scan from the medical scan. An inference function3030 can be performed on the pre-processed medical scan by utilizing thecomputer vision model generated as discussed in FIG. 12 to produceinference data. The inference data can be transmitted to a client device120 for display via a display device, for example, by utilizing themedical scan assisted review system 102. Alternatively or in addition,the inference data can be transmitted back to the medical scan database342 to be mapped to the medical scan and/or can be transmitted to one ormore other subsystems 101.

The pre-processing step 3006 can the same a pre-processing step 3005. Insome embodiments, pre-processing step 3006 includes applying a singleintensity transformation function to the medical scan to produce asingle augmented image. In some embodiments, pre-processing step 3006includes selecting the single intensity transformation function from setof intensity transformation functions utilized in pre-processing step3005, and/or otherwise determining the parameters of the singleintensity transformation function. For example, the pre-processing stepcan include determining the single intensity transformation functionbased on the scan classifier data 420, modality of the medical scan, theanatomical region of the medical scan, the patient history or otherpatient data corresponding to the medical scan, longitudinal datareceived along with the medical scan, metadata of the medical scan,density properties of an a anatomical feature included in the medicalscan, density properties of an anatomical feature identified as being ofinterest, and/or a density value histogram of the medical scan. In someembodiments, the pre-processing step 3006 includes selecting theintensity transformation function based on the random intensitytransformation function parameter selection as discussed herein. In someembodiments, the pre-processing step 3006 includes selecting theintensity transformation function based on the density value histogramdetermined for the image data of the new medical scan and/or based onmost similar one of the pre-determined grouping generated based on thescan classifier data, as discussed herein, to generate an augmentedimage with a desired output distribution. In some embodiments, thepre-processing step 3006 includes determining to apply an identityfunction and/or includes otherwise leaving the image data of the medicalscan unaltered.

The transformation function parameter data utilized by thepre-processing step 3006 can be the same as the transformation functionparameter data of pre-processing step 3005, can correspond to a subsetof the transformation function parameter data of pre-processing step3005, for example, corresponding to a single one of the set of intensitytransformation functions, and/or can be different from thetransformation function parameter data of pre-processing step 3005. Thetransformation function parameter data can be stored locally, can befetched from the medical scan analysis function database 346 as shown inFIG. 12G, and/or can otherwise be determined.

The inference function 3030 can be performed by utilizing the computervision model indicated in the model data to produce inference data. Theinference function can utilize the inference step 1354 of FIG. 7B togenerate one or more probability matrices 1371 of the inference data.The inference function 3030 can utilize the detection step 1372,abnormality classification step 1374, similar scan identification step1376, and/or display parameter step 1378 of FIG. 7B in generating theinference data. The display parameter step 1378 can indicate contrastparameters corresponding to the intensity transformation functionutilized in performing the pre-processing step 3006. The inference datacan correspond to some or all diagnosis data 440 and/or can indicate oneor more abnormalities detected in the medical scan. The inferencefunction can correspond to any medical scan analysis function and/orinference function described herein. The model data utilized to performthe inference function can be retrieved from the medical scan analysisfunction database 346 and/or can be stored locally.

In some embodiments, a subset of the set of intensity transformationfunctions of size greater than two, are selected in the pre-processingstep 3006 in the same fashion as selection in pre-processing step 3005.In some embodiments, the entire set of intensity transformationfunctions are utilized. The subset or entire set of intensitytransformation functions can be applied to the new medical scan toproduce a plurality of augmented images. In some embodiments, each ofthe augmented images are fed into the model separately, and thusinference data is generated for each of the plurality of augmentedimages. The inference function 3030 can include processing this set ofinference data to determine final inference data, for example, bydetermining consensus inference data across the set of inference data,by performing an average on some or all of the set of inference data, byindicating all of the set of inference data, by indicating differentabnormalities indicating in different ones of the set of inference data,by selecting a most severe one of the set of inference data and/or oneof the set of inference data indicating a highest probability of one ormore abnormalities, and/or by otherwise consolidating the set ofinference data.

In other embodiments, performing the training step 3010 includes jointlyprocessing the set of augmented images for each medical scan in thetraining set as a same set of input data to the model. A set ofaugmented images for the new medical scan can be generated by performingthe same set of intensity transformation functions in pre-processingstep 3006, for example, by utilizing the same pre-processing step 3005.Performing the inference function 3030 can include jointly processingthis set of augmented images for the new medical scan by feeding the setof augmented images as input to the model to generate the inferencedata.

FIG. 12H illustrates an embodiment of an intensity transformaugmentation system 3002 that is utilized as a contrast parameterlearning system. Rather than utilizing a pre-determined set of intensitytransformation functions and/or random intensity transformationfunctions as discussed herein, the intensity transform augmentationsystem 3002 can learn and/or otherwise generate one or more intensitytransformation functions to utilize in pre-processing the training datato generate the model and/or in pre-processing new medical scans thatare fed into the model in generating inference data.

As shown in FIG. 12H, a transform parameter evaluation step 3070utilizes model data of a computer vision model to determine significanttransformation function parameter data. The significant transformationfunction parameter data can correspond to one or more intensitytransformation function parameters and/or other contrast parameter datadetermined to be statistically significant, determined to favorablyinfluence the model accuracy, and/or otherwise determined to be utilizedto re-train the model to improve the performance of the model. Thepre-processing step 3005 can be utilized to generate a pre-processedtraining set by utilizing the transform parameter data on a medical scantraining set, which can be the same or different from the medical scantraining set originally used to train the computer vision model. Thepre-processed training set can correspond to a re-contrasted version ofthe training set utilized to train the computer vision model. Thepre-processing step 3005 can include performing one or more intensitytransformation functions indicated in the significant transformationfunction parameter data determined in the transform parameter evaluationstep 3070 to generate the pre-processed training set, as discussed inconjunction with FIG. 12A. Training step 3010 can be performed byutilizing the pre-processing training set as discussed in conjunctionwith FIG. 12A to generate an updated computer vision model. The modeldata of the updated computer vision model can be sent to medical scanfunction analysis database 346 as a new entry, and/or a new version orotherwise or an updated entry of the previous model. Alternatively or inaddition, the significant transformation function parameters can be sentto the medical scan function analysis database 342 as one or more newentries, as one or more new version of one or more existing entries forprevious transformation function parameters, and/or as an updated entryof one or more existing entries for previous transformation functionparameters.

The updated model data can be utilized by the intensity transformaugmentation system itself and/or by other subsystems 101 to generateinference data for new medical scans as discussed in conjunction withFIG. 12G. In some embodiments the significant transformation functionparameter data determined in transform parameter evaluation step 3070can be utilized in performing pre-processing step 3006 on the newmedical scans before applying the updated model to generate apre-processed new medical scan. The pre-processed new medical scan cancorrespond to a re-contrasted version of the new medical scan received.An updated inference function corresponding to the updated computervision model can be applied to the pre-processed new medical scan togenerate inference data for transmission to a client device as discussedin conjunction with FIG. 12G.

In some embodiments, the significant transformation function parameterdata indicates different transformation function parameter data and/or adifferent one of the set of intensity transformation functions fordifferent sets of medical scan input criteria, such as scan classifierdata and/or density value histograms as discussed herein. A singleintensity transformation function can be selected based on correspondingfeatures of new medical scan as discussed in conjunction with FIG. 12G.In such embodiments, the significance of this criteria and/or thesegmentation of different groupings of medical scan features utilized todetermine which one or more intensity transformation function will beutilized on new medical scans can also be generated in the performanceof the transform parameter evaluation step 3070.

The model data can correspond to computer vision model generated by theintensity transform augmentation system 3002 itself, for example, inaccordance with FIG. 12A. In such cases, the model data can be retrievedfrom local memory. The model data can correspond to a computer visionmodel generated by utilizing medical scan image analysis system 112,and/or can correspond to any computer vision model trained on a trainingset of medical scans. The computer vision model indicated by the modeldata can have been generated with or without performing a pre-processingstep on the a training set of medical scans. In some embodiments, atleast one of the set of intensity transformation functions as performedon each medical scan in the training set to produce a pre-processedtraining set as discussed in conjunction with FIG. 12A. In someembodiments, every one of the set of intensity transformation functionsas performed on each medical scan in the training set to produce apre-processed training set. In some embodiments, the set of intensitytransformation functions are selected as a test set of intensitytransformation functions and/or an initial set of intensitytransformation functions, for example, selected based on user input to aclient device for display via a display device.

In some embodiments, performing the transform parameter evaluation step3070 includes generating a plurality of significance values for each ofthe set of intensity transformation functions based on evaluating eachof their effect on the model. In some embodiments, the set ofsignificance values are ranked, where one or more of the set ofintensity transformation functions with a highest ranking are indicatedin the significant transformation function parameter data. In someembodiments, the significance values are compared to a significancethreshold. For example, the significance threshold can be determinedbased on user input to client device 120 via interaction with a userinterface displayed on a display device. Ones of the set of intensitytransformation functions with significance values that compare favorablyto the significance threshold can be indicated in the significanttransformation function parameter data.

In some embodiments, the means of determining the density window data475 and/or display parameters 470 as discussed in conjunction with themedical scan image analysis system can be utilized in performing thetransform parameter evaluation step 3070. The density window boundarypair (or alternatively, density window center and density window width)of the determined density window and/or contrast parameters indicated inthe display parameters can be indicated in the significanttransformation function parameter data.

In some embodiments, the model data indicates a plurality of computervision models. For example, a plurality of computer vision models can begenerated by performing a different pre-processing step 3005 one thesame training set of medical scans. Each one of the computer visionmodels can correspond to a pre-processed training set that correspondsto one or more augmented images for each one of the medical scans byutilizing different intensity transformation functions. Some or all ofthe plurality of computer vision models can be generated by theintensity transform augmentation system 3002 as discussed in conjunctionwith FIG. 12A.

In some embodiments, the set of computer vision models includes Ncomputer vision models, where each of the set of N computer visionmodels corresponds to a pre-processed training data that includesaugmented images generated by utilizing a single, corresponding one of Ndifferent intensity transformation functions on the training data. Insuch embodiments, determining the significance values can includegenerating accuracy data based on performing an accuracy step on eachcomputer vision model, for example, where test data is fed into themodel and the inference data output is compared to known output labelsof the test data to generate the accuracy data. Thus, a model with themost favorable accuracy data can indicate that the corresponding one ofthe set of intensity transformation functions is of most relativesignificance, and/or the that the corresponding one of the set ofintensity transformation functions can be assigned a highestsignificance value. In some embodiments, the significance thresholdcorresponds to an accuracy threshold to which the accuracy data of thecorresponding model is compared.

Conversely, each of the set of N computer vision models can correspondto a pre-processed training data that includes augmented imagesgenerated by utilizing each of the N possible combinations of N−1different intensity transformation functions on the training data. Thus,all but one of the set of intensity transformation functions areutilized to generate augmented images utilized in the training set ofeach model. Accuracy data can similarly be generated for each of thesemodels. In this case, least favorable accuracy data can indicate thatthe corresponding one of the set of intensity transformation functionsthat was not included is of most relative significance.

In some embodiments, accuracy differentials between some or all of themodels can further indicate whether any of the intensity transformationfunctions are significant. In some embodiments, a control computervision model can be generated by utilizing a pre-processed training set,for example where the pre-processed training set is generated byperforming all of the set of intensity transformation functions on eachof the set of training data. Alternatively or in addition, a controlcomputer vision model can be generated by performing only an identityfunction, selecting random contrast parameters, and/or otherwise leavingthe medical scan image data unaltered in generating the training setutilized to train the model. Alternatively or in addition, a controlcomputer vision model can be generated by utilizing a pre-processedtraining set generated by utilizing previously determined significanttransformation function parameters, for example, as part of an iterativeprocess to further improve performance. Some or all of the additionalcomputer vision models generated by intelligently selecting subsets ofthe set of intensity transformation functions, and/or by determiningrandom and/or new parameters for a new set of intensity transformationfunctions, can be considered experimental computer vision models.Accuracy data for the experimental computer vision models can becompared to one or more of the control computer vision models, and/orcan be compared to each other. The magnitude of differentials betweenaccuracy data between experimental computer vision models and the one ormore control computer vision models can be utilized to determine whetherintensity transformation functions that are included and/or that wereremoved from each experimental computer vision model are statisticallysignificant and/or otherwise influence the model. Experimental computervision models with accuracy data with highest magnitude differentialsfrom the control can indicate ones of the set of intensitytransformation functions that where included and/or removed that have agreatest influence on the model.

In some embodiments, the transform parameter evaluation step 3070includes an iterative process of tweaking parameters of one or moreintensity transformation functions. For example, random orpre-determined parameters for one or more initial intensitytransformation functions utilized to pre-process training data isdetermined. One or more models are generated and evaluated, and one ormore of the parameters are changed accordingly, for example, wherechanges are greater when the parameters are determined to be lesssignificant and when changes are smaller when the parameters aredetermined to be more significant. Alternatively or in addition, one ormore intensity transformation functions are added and/or dropped fromthe set of intensity transformation functions based on which of the setof intensity transformation functions are determined to be significant.One or models are then regenerated by utilizing the new parameters, andthe new parameters are again changed accordingly and/or one or moreintensity transformation functions are added and/or dropped from the setof intensity transformation functions accordingly. This process cancontinue for a fixed number of iterations, until the accuracy data ofthe one or more models compares favorably to an accuracy threshold,and/or until the significance score of one or more of the intensitytransformation functions compares favorably to a threshold. In someembodiments, this iterative process is in accordance with anoptimization algorithm and/or a greedy optimization algorithm.

In some embodiments, the transform parameter evaluation step 3070 isinherently incorporated within the training process of the model itself.For example, the model can take density value and/or raw sensor data ofthe medical scan in an input layer of a neural network, and/or can takegreyscale pixel values corresponding to original contrast parameters. Afirst layer and/or first set of layers can correspond to thepre-processing step, where weights connecting the input layer and firstset of layers correspond to performing the intensity transformationfunction. For example, the weights can indicate the deterministicmapping. A final one of the first set of layers of the neural networkcan correspond to the pixel values of the augmented image data and/or anotherwise re-contrasted image. Performing the training step 3010 totrain the model is utilized to generate the final set of weightscorresponding to the learned intensity transformation function, forexample, where these weights of the first set of layers are iterativelyimproved to increase accuracy of the model. In some embodiments, aweights for a plurality of intensity transformation functions arelearned in this process, and criteria for which types of medical scanseach of the plurality of intensity transformation functions are appliedcan be learned and/or can be predesignated. In some embodiments, othermodel types and/or learning methodologies are employed to determine theintensity transformation function and/or to learn a plurality ofintensity transformation function to be applied to medical scans thatfit a set of predesignated or learned criteria.

In some embodiments, the significant transformation function parameterdata indicates that no transformation function parameters weredetermined to be significant, indicates that no intensity transformationfunctions out-performed the others, and/or otherwise indicates that themodel accuracy is not affected by re-contrasting and/or other intensitytransformations. In such embodiments, the intensity transformaugmentation system 3002 can determine to keep the original model data,and the re-training is not performed. Similarly, image data of newincoming medical scans can remain unaltered when the inference functionis performed. In some embodiments, in response to determining thataltering contrast parameters does not significantly affect the model,the model can be re-trained on the original, raw input data, forexample, if it had been previously trained on a different image formatcorresponding to applying contrast parameters to the raw input data. Insome embodiments, in response to determining significant transformationfunction parameter data indicates that no transformation functionparameters were determined to be significant, the iterative process oftweaking parameters can be initiated until significant transformationfunction parameters are determined. This can include re-setting initialparameters to new random initial parameters, prompting the user to setnew initial parameters via user input to a user interface, and/ortweaking the current parameters based on their respective performance.

In some embodiments, the intensity transform augmentation system 3002utilizes a plurality of computer vision models. In particular theplurality of computer vision models can correspond to different medicalscan types, different medical scan modalities, different anatomicalregions, different patient history, different medical entities,different geographic regions, different types of abnormalities that aredetected, and/or other different criteria. In some embodiments, thecriteria segmenting the plurality of different computer vision models isbased on the scan classifier data 420. In some embodiments, the criteriasegmenting the plurality of different computer vision models isdetermined based on user input to client device 120 via interaction witha user interface displayed on a display device. In some embodiments, thecriteria segmenting the plurality of different computer vision models islearned by the intensity transform augmentation system 3002, forexample, in conjunction with performing the transform parameterevaluation step 3070. The criteria distinguishing each of the pluralityof computer vision models can be utilized to identify correspondingtraining data utilized to train each model. Furthermore, one of theplurality of computer vision models can be selected to process incomingmedical scans based on evaluating the medical scan and/or metadata ofthe medical scan and determining which of the plurality of computervision models has corresponding criteria that most favorably compares tothe medical scan.

In such embodiments, the training data of each of the plurality ofcomputer vision models can undergo separate pre-processing steps 3005that utilize different sets of intensity transformation functions and/ordifferent transformation function parameter data corresponding to thesame or different types of intensity transformation functions. In someembodiments, set of intensity transformation functions utilized inpre-processing steps 3005 for each of the plurality of computer visionmodels can be determined based on user input to client device 120 viainteraction with a user interface displayed on a display device.

In some embodiments, the set of intensity transformation functionsutilized in pre-processing steps 3005 can be determined based on densityproperties of anatomical features corresponding to the scan type and/oranatomical region of the corresponding one of the plurality of computervision models. In some embodiments, the set of intensity transformationfunctions utilized in pre-processing steps 3005 can be determined basedon density properties of types of abnormalities that are detected in thescan type and/or anatomical region of the corresponding one of theplurality of computer vision models. In some embodiments, the set ofintensity transformation functions utilized in pre-processing steps 3005can be determined based on density properties of one or more particulartypes of abnormalities that the corresponding one of the plurality ofcomputer vision models is trained to detect. In some embodiments,different random distributions corresponding to random selection ofintensity transformation function parameters are utilized by thedifferent ones of the plurality of computer vision models.

In some embodiments, the set of intensity transformation functionsutilized in pre-processing steps 3005 for each of the plurality ofcomputer vision models are determined in accordance with the transformparameter evaluation step 3070 of FIG. 12H. For example, each of theplurality of computer vision models can be processed separately, anddifferent significant transformation function parameter data can begenerated for each of the plurality of computer vision models, anddetermine the indicate the set of intensity transformation functionsutilized in pre-processing steps 3005. In some embodiments, determiningthe criteria that segment different ones of the of computer visionmodels can also be determined by utilizing the transform parameterevaluation step, for example, as part of an iterative process.

In various embodiments, the intensity transform augmentation system 3002includes a at least one processor and a memory that stores operationalinstructions that, when executed by the at least one processor, causethe intensity transform augmentation system to receive, via a receiver,a training set of medical scans. A plurality of sets of augmented imagesare generated, where each set of augmented images in the plurality ofsets of augmented images is generated by performing a set of intensitytransformation functions on one of the training set of medical scans,and where each of the set of intensity transformation functions arebased on density properties of corresponding one of a plurality ofdifferent anatomy features present in the training set of medical scans.A computer vision model is generated by performing a training step onthe plurality of sets of augmented images, where each augmented image ofa set of augmented images is assigned same output label data based on acorresponding one of the training set of medical scans. A new medicalscan is received, via the receiver. Inference data is generated byperforming an inference function that utilizes the computer vision modelon the new medical scan. The inference data is transmitted, via atransmitter, to a client device for display via a display device.

Alternatively or in addition, the operational instructions, whenexecuted by the at least one processor, cause the intensity transformaugmentation system to receive, via the receiver, a training set ofmedical scans. Random intensity transformation function parameters aregenerated for each medical scan of the training set of medical scans. Aplurality of augmented images are generated, where each of the pluralityof augmented images is generated by performing a intensitytransformation function on one of the training set of medical scans byutilizing the random intensity transform parameters generated for theone of the training set of medical scan. A computer vision model isgenerated by performing a training step on the plurality of augmentedimages. A new medical scan is received via the receiver. Inference datais generated by performing an inference function that utilizes thecomputer vision model on the new medical scan. The inference data istransmitted, via the transmitter, to a client device for display via adisplay device.

Alternatively or in addition, the intensity transform augmentationsystem is implemented as a contrast parameter learning system. Theoperational instructions, when executed by the at least one processor,cause the contrast parameter learning system to generate contrastsignificance data for a computer vision model, where the computer visionmodel was generated by performing a training step on a training set ofmedical scans. Significant contrast parameters are identified based onthe contrast significance data. A re-contrasted training set isgenerated by performing an intensity transformation function on thetraining set of medical scans, where the intensity transformationfunction utilizes the significant contrast parameters. A re-trainedmodel is generated by performing the training step on the firstre-contrasted training set. A new medical scan is received via thereceiver. Re-contrasted image data of the new medical scan is generatedby performing the intensity transformation function. Inference data isgenerated by performing an inference function that utilizes the firstre-trained model on the re-contrasted image data. The inference data istransmitted via the transmitter to a client device for display via adisplay device.

FIG. 13 presents a flowchart illustrating a method for execution by anintensity transform augmentation system 3002 that stores executionalinstructions that, when executed by at least one processor, cause theintensity transform augmentation system 3002 to perform the steps below.

Step 3102 includes receiving, via a receiver, a training set of medicalscans. Step 3104 includes generating a plurality of sets of augmentedimages. Each set of augmented images in the plurality of sets ofaugmented images is generated by performing a set of intensitytransformation functions on one of the training set of medical scans.Each of the set of intensity transformation functions can be based ondensity properties of corresponding one of a plurality of differentanatomy features present in the training set of medical scans. Forexample, the plurality of different anatomy features can include bone,soft tissue, fat, fluid, and/or air.

Step 3106 includes generating a computer vision model by performing atraining step on the plurality of sets of augmented images. Eachaugmented image of the set of augmented images is assigned same outputlabel data based on a corresponding one of the training set of medicalscans. Step 3108 includes receiving, via the receiver, a new medicalscan. Step 3110 includes generating inference data by performing aninference function that utilizes the computer vision model on the newmedical scan. Step 3112 includes transmitting, via a transmitter, theinference data to a client device for display via a display device.

In various embodiments, performing one of the set of intensitytransformation functions on the one of the training set of the medicalscans includes adjusting a contrast of image data of the one of thetraining set of medical scans to accentuate the corresponding one of theplurality of different anatomy features. In various embodiments,performing one of the set of intensity transformation functions on theone of the training set of the medical scans includes adjusting abrightness of image data of the one of the training set of medical scansto accentuate the corresponding one of the plurality of differentanatomy features.

In various embodiments, the training set of medical scans includes afirst subset of medical scans that corresponds to a first anatomicalregion and a second subset of medical scans that corresponds to a secondanatomical region. A first proper subset of the set of intensitytransformation functions is applied to medical scans of the first subsetbased on a first subset of the plurality of different anatomy featurespresent in the first anatomical region. A second proper subset of theset of intensity transformation functions is applied to medical scans ofthe second subset based a second subset of the plurality of differentanatomy features present in the second anatomical region. A setdifference between the first proper subset of the set of intensitytransformation functions and the second proper subset of the set ofintensity transformation functions is non-null.

In various embodiments, the training set of medical scans includes afirst subset of medical scans that corresponds to a first abnormalityoutput label and a second subset of medical scans that corresponds to asecond abnormality output label. A first proper subset of the set ofintensity transformation functions is applied to medical scans of thefirst subset based on first density properties of the first abnormalityoutput label. A second proper subset of the set of intensitytransformation functions is applied to medical scans of the secondsubset based on second density properties of the second abnormalityoutput label. A set difference between the first proper subset of theset of intensity transformation functions and the second proper subsetof the set of intensity transformation functions is non-null.

In various embodiments, the training set of medical scans includes afirst subset of medical scans that corresponds to a first region ofinterest output label and a second subset of medical scans thatcorresponds to a second region of interest output label. A first propersubset of the set of intensity transformation functions is applied tomedical scans of the first subset based on a first subset of theplurality of different anatomy features that correspond to the firstregion of interest output label, and a second proper subset of the setof intensity transformation functions is applied to medical scans of thesecond subset based on a second subset of the plurality of differentanatomy features that correspond to the second region of interest outputlabel. A set difference between the first proper subset of the set ofintensity transformation functions and the second proper subset of theset of intensity transformation functions is non-null.

In various embodiments, the set of intensity transformation functions isperformed on image data of the training set of medical scans to producethe plurality of sets of augmented images. Generating the computervision model includes performing the training step on the image data ofthe training set of medical scans and the plurality of sets of augmentedimages. In various embodiments, each of the set of intensitytransformation functions is performed on raw sensor data of one of thetraining set of medical scans to generate one of a corresponding set ofaugmented images. In various embodiments, the raw sensor data is in afirst data format and the one of the corresponding set of augmentedimages is in a second data format. In various embodiments, the seconddata format is a JPEG image format.

In various embodiments, performing each of the set of intensitytransformation functions includes calculating each one of a plurality ofpixel values of the augmented image as a deterministic function of eachcorresponding one of a plurality of density values of the raw sensordata. In various embodiments, the deterministic function utilized by atleast one of the set of intensity transformation functions is anon-linear function of density value. In various embodiments, some orall of the intensity transformation functions utilize a deterministicfunction of density value in accordance with a different mean anddifferent standard deviation of a cumulative distribution function of anormal distribution.

In various embodiments, some or all of the set of intensitytransformation functions is in accordance with a different densitywindow boundary pair. In various embodiments, some or all of the set ofintensity transformation functions is in accordance with a differentdensity window center. In various embodiments, some or all of the set ofintensity transformation functions is in accordance with a differentdensity window width.

In various embodiments, function parameter data is received from aclient device. The function parameter data corresponds to user input inresponse to a prompt displayed via a user interface on a display devicethat corresponds to the client device. At least one of the set ofintensity transformation functions is performed in accordance with thefunction parameter data.

In various embodiments, the same output label data indicates knownabnormality data of the corresponding one of the training set of medicalscans, and the inference data indicates at least one abnormalitydetected in the new medical scan.

In various embodiments, each medical scan of the training set of medicalscans includes a set of image slices. The set of intensitytransformation functions is performed on each of the set of image sliceson one of the training set of medical scans to generate a correspondingset of augmented images of the plurality of sets of augmented images. Invarious embodiments, the training set of medical scans includes aplurality of longitudinal data sets that each include a plurality ofmedical scans. The set of intensity transformation functions isperformed on each of the plurality of longitudinal data sets to generatea corresponding longitudinal set of augmented images of the plurality ofsets of augmented images.

FIG. 14 presents a flowchart illustrating a method for execution by anintensity transform augmentation system 3002 that stores executionalinstructions that, when executed by at least one processor, cause theintensity transform augmentation system 3002 to perform the steps below.Some or all steps of FIG. 14 can be performed by the intensity transformaugmentation system 3002 alternatively or in addition to some or all ofthe steps of FIG. 13 .

Step 3202 includes receiving, via a receiver, a training set of medicalscans. Step 3204 includes generating random intensity transformationfunction parameters for each medical scan of the training set of medicalscans. Step 3206 includes generating a plurality of augmented images,where each of the plurality of augmented images is generated byperforming a intensity transformation function on one of the trainingset of medical scans by utilizing the random intensity transformationparameters generated for the one of the training set of medical scans.Step 3208 includes generating a computer vision model by performing atraining step on the plurality of augmented images. Step 3210 includesreceiving, via the receiver, a new medical scan. Step 3212 includesgenerating inference data by performing an inference function thatutilizes the computer vision model on the new medical scan. Step 3214includes transmitting, via a transmitter, the inference data to a clientdevice for display via a display device.

In various embodiments, generating the computer vision model includesperforming the training step on image data of each of the training setof medical scans and the plurality of augmented images.

In various embodiments, performing the intensity transformation functionon the one of the training set of medical scans includes calculatingeach one of a plurality of greyscale pixel values of the augmented imageas a deterministic function of each corresponding one of a plurality ofdensity values of raw sensor data of the one of the training set ofmedical scans. In various embodiments, performing the intensitytransformation function on the one of the training set of medical scansincludes calculating each one of a final plurality of greyscale pixelvalues of the augmented image as a deterministic function of eachcorresponding one of an initial plurality of greyscale pixel values ofimage data of the one of the training set of medical scans. In variousembodiments, generating the computer vision model includes performingthe training step on image data of each the training set of medicalscans and the plurality of augmented images.

In various embodiments, generating the random intensity transformparameters includes randomly selecting a density window center and adensity window boundary pair. The intensity transformation function is adensity windowing function defined by the density window center and thedensity window boundary pair. In various embodiments, the randomintensity transformation function is a non-linear function defined bythe random intensity transform parameters. In various embodiments,generating the random intensity transform parameters includes randomlyselecting a mean value and further includes randomly selecting astandard deviation value. The intensity transformation function isperformed in accordance with the mean value and the standard deviationvalue. In various embodiments, performing the intensity transformationfunction on the one of the training set of medical scans includescalculating each one of a final plurality of pixel values of theaugmented image as a deterministic function of each corresponding one ofan initial plurality of pixel values of the one of the training set ofmedical scans, where the deterministic function is a cumulativedistribution function of a normal distribution designated by the meanvalue and the standard deviation value.

In various embodiments, a final histogram indicating an occurrence ofpixel values for each of the plurality of augmented images reflects themean value and the standard deviation value. In various embodiments, aninitial histogram indicating an occurrence of pixel values for the oneof the training set of medical scans is utilized to determine the randomintensity transformation function such that the final histogram reflectsthe mean value and the standard deviation value.

In various embodiments, the random intensity transformation functionparameters are selected randomly from a discrete set of pre-determinedparameter options. In various embodiments, the random intensitytransformation function parameters are generated in accordance with anon-uniform random distribution. In various embodiments, an optimalrandom distribution for parameter selection is determined. The randomintensity transformation function parameters for all of the training setof medical scans are generated in accordance with the optimal randomdistribution. In various embodiments random distribution parameter datais received from a client device. The random distribution parameter datacorresponds to user input in response to a prompt displayed via a userinterface on a display device that corresponds to the client device. Thenon-uniform random distribution utilized to generate the randomintensity transformation function parameters corresponds to the randomdistribution parameter data.

In various embodiments, a set of random distributions for parameterselection are determined. selecting one of the set of randomdistributions is selected for each medical scan of the training set ofmedical scans, based on a metadata or other characteristics of eachmedical scan of the training set of medical scans. The random intensitytransformation function parameters are generated for the correspondingmedical scan in accordance with the one of the set of randomdistributions.

In various embodiments, a set of augmented images are generated for eachof the training set of medical scans, and the random intensitytransformation function parameters are generated separately forutilization in performing the intensity transformation function on theeach of the training set of medical scans to generate each of the set ofaugmented images. In various embodiments a set of random distributionsfor parameter selection are determined. The random intensitytransformation function parameters are generated in accordance with eachof the set of random distributions for the each of the training set ofmedical scans to generate the set of augmented images. In variousembodiments, the set of random distributions each have mean parametervalues corresponding to one of a set of windowing functions. In variousembodiments, each of the set of windowing functions is based on densityproperties of a corresponding one of a plurality of different anatomyfeatures.

In various embodiments, a set of output label data corresponding to thetraining set of medical scans indicates known abnormality data of thetraining set of medical scans, performance of the training step utilizesthe set of output label data, and the inference data indicates at leastone abnormality detected in the new medical scan.

FIG. 15 presents a flowchart illustrating a method for execution by anintensity transform augmentation system 3002 for example, implemented asa contrast parameter learning system. The intensity transformaugmentation system 3002 stores executional instructions that, whenexecuted by at least one processor, cause the intensity transformaugmentation system 3002 to perform the steps below. Some or all stepsof FIG. 15 can be performed by the intensity transform augmentationsystem 3002 alternatively or in addition to some or all of the steps ofFIG. 13 and/or FIG. 14 .

Step 3302 includes generating first contrast significance data for afirst computer vision model, where the computer vision model wasgenerated by performing a training step on a first training set ofmedical scans. Step 3304 includes identifying first significant contrastparameters based on the first contrast significance data. Step 3306includes generating a first re-contrasted training set by performing afirst intensity transformation function on the first training set ofmedical scans, where the first intensity transformation functionutilizes the first significant contrast parameters. Step 3308 includesgenerating a first re-trained model by performing the training step onthe first re-contrasted training set. Step 3310 includes receiving, viaa receiver, a new medical scan. Step 3312 includes generatingre-contrasted image data of the new medical scan by performing the firstintensity transformation function. Step 3314 includes generatinginference data by performing an inference function that utilizes thefirst re-trained model on the re-contrasted image data. Step 3316includes transmitting, via a transmitter, the inference data to a clientdevice for display via a display device.

In various embodiments, the first training set of medical scanscorresponds to a first one of a plurality of medical scan types. Secondcontrast significance data is generated for a second computer visionmodel, where the second computer vision model was generated byperforming the training step on a second training set of medical scansthat correspond to a second one of a plurality of medical scan typesthat is different from the first one of the plurality of medical scantypes. Second significant contrast parameters are identified based onthe second contrast significance data, where the second significantcontrast parameters are different from the first significant contrastparameters. A second re-contrasted training set is generated byperforming a second intensity transformation function on a the secondtraining set of medical scans, where the second intensity transformationfunction utilizes the second significant contrast parameters. A secondre-trained model is generated by performing the training step on thesecond re-contrasted training set. The first intensity transformationfunction and the inference function that utilizes the first re-trainedmodel is utilized for the new medical scans in response to determiningthe new medical scan corresponds to the first one of the plurality ofmedical scan types.

In various embodiments, the first one of the plurality of medical scantypes and the second one of the plurality of medical scan typescorrespond to different anatomical regions. In various embodiments, thefirst one of the plurality of medical scan types and the second one ofthe plurality of medical scan types correspond to different modalities.

In various embodiments, first intensity transformation function is anon-linear function. In various embodiments, the first significantcontrast parameters indicate a mean and a standard deviation utilized toperform the non-linear function on each of a plurality of pixels toproduce re-contrasted image data. In various embodiments, the non-linearfunction is a CDF corresponding to a normal distribution. In variousembodiments, the first significant contrast parameters indicate adensity window boundary pair.

In various embodiments a plurality of sets of augmented images aregenerated. Each set of augmented images in the plurality of sets ofaugmented images is generated by performing a set of intensitytransformation functions on one of the first training set of medicalscans. In various embodiments, each of the set of intensitytransformation functions are based on density properties ofcorresponding one of a plurality of different anatomy features presentin the training set of medical scans. The first computer vision model isgenerated by performing the training step on the plurality of sets ofaugmented images, where each augmented image of a set of augmentedimages is assigned same output label data based on a corresponding oneof the training set of medical scans. In various embodiments,significant contrast parameters utilized to perform the first intensitytransformation function correspond to contrast parameters utilized by acorresponding one of the set of intensity transformation functions. Invarious embodiments, the first contrast significance data indicates aranking of the set of intensity transformation functions, and the firstsignificant contrast parameters are identified to correspond to thecontrast parameters utilized by the one of the set of intensitytransformation functions in response to the one of the set of intensitytransformation functions having a most favorable rank in the ranking.

In various embodiments, the first contrast significance data indicates aproper subset of the set of intensity transformation functions thatincludes at least two of the set of intensity transformation functions.The first re-contrasted training set is generated by performing a theproper subset of intensity transformation function on the first trainingset of medical scans to generate a plurality of sets of augmentedimages. The first re-trained model is generated by performing thetraining step on the plurality of sets of augmented images. In variousembodiments, significance values are calculated for each of the set ofintensity transformation functions. A significant significant subset ofthe set of intensity transformation functions is generated by includingones of the set of intensity transformation functions with acorresponding significance value that compares favorably to asignificance threshold. The proper subset corresponds to the significantsubset.

In various embodiments, the re-contrasted image data of the new medicalscan includes a set of images, generated by performing the proper subsetof the set of intensity transformation functions on the new medicalscan. Generating the inference data includes performing the inferencefunction on the set of images the re-contrasted image data to generate aset of partial inference data. A consensus function is performed on theset of partial inference data to generate the inference data.

In various embodiments, a plurality of sets of augmented images aregenerated. Each set of augmented images in the plurality of sets ofaugmented images is generated by performing a set of intensitytransformation functions on one of the first training set of medicalscans. A set of computer vision models are generated that eachcorrespond to one of the set of intensity transformation functions. Eachof the set of computer vision models is generated by performing thetraining step on ones of the plurality of sets of augmented images thatwere generated by utilizing the corresponding one of the set of theintensity transformation functions. Model accuracy data is generated foreach of the set of computer vision models. Model ranking data isgenerated by ranking the set of computer vision models in accordancewith the model accuracy data. The first contrast significance dataindicates the one of the set of intensity transformation functions thatcorresponds to a most favorably ranked one of the set of computer visionmodels in the model ranking data.

In various embodiments, the first contrast significance data isgenerated by calculating significance values for each of a set ofcontrast parameters that each correspond to contrast settings for atleast one of the medical scans of the first training set. Ones of theset of contrast parameters with a corresponding significance value thatcompares favorably to a significance threshold are indicated in thefirst contrast significance data. In various embodiments, none of theset of contrast parameters have a corresponding significance value thatcompares favorably to the significance threshold, and where the firstintensity transformation function corresponds to an identity function inresponse to the first contrast significance data indicating nosignificant contrast parameters.

In various embodiments, the set of contrast parameters correspond to aset of intensity transformation functions applied to the medical scansof the training set, and none of the set of contrast parameters have acorresponding significance value that compares favorably to thesignificance threshold. A new set of contrast parameters are selected.An updating training set of medical scans are generated by performing anew set of intensity transformation functions that utilize the new setof contrast parameters on the training set of medical scans. Thetraining step is re-performed on a the updated training set of medicalscans to generate an updated first computer vision model. Updated firstcontrast significance data is generated for the updated first computervision model, where the first significant contrast parameters areidentified based on the updated first contrast significance data, andwhere the first significant contrast parameters include contrastparameters of the new set of contrast parameters.

As may be used herein, the terms “substantially” and “approximately”provides an industry-accepted tolerance for its corresponding termand/or relativity between items. Such an industry-accepted toleranceranges from less than one percent to fifty percent and corresponds to,but is not limited to, component values, integrated circuit processvariations, temperature variations, rise and fall times, and/or thermalnoise. Such relativity between items ranges from a difference of a fewpercent to magnitude differences. As may also be used herein, theterm(s) “configured to”, “operably coupled to”, “coupled to”, and/or“coupling” includes direct coupling between items and/or indirectcoupling between items via an intervening item (e.g., an item includes,but is not limited to, a component, an element, a circuit, and/or amodule) where, for an example of indirect coupling, the intervening itemdoes not modify the information of a signal but may adjust its currentlevel, voltage level, and/or power level. As may further be used herein,inferred coupling (i.e., where one element is coupled to another elementby inference) includes direct and indirect coupling between two items inthe same manner as “coupled to”. As may even further be used herein, theterm “configured to”, “operable to”, “coupled to”, or “operably coupledto” indicates that an item includes one or more of power connections,input(s), output(s), etc., to perform, when activated, one or more itscorresponding functions and may further include inferred coupling to oneor more other items. As may further be used herein, the term “associatedwith”, includes direct and/or indirect coupling of separate items and/orone item being embedded within another item. As may still further beused herein, the term “automatically” refers to an action causeddirectly by a processor of a computer network in response to atriggering event and particularly without human interaction.

As may be used herein, the term “compares favorably”, indicates that acomparison between two or more items, signals, etc., provides a desiredrelationship. For example, when the desired relationship is that signal1 has a greater magnitude than signal 2, a favorable comparison may beachieved when the magnitude of signal 1 is greater than that of signal 2or when the magnitude of signal 2 is less than that of signal 1. As maybe used herein, the term “compares unfavorably”, indicates that acomparison between two or more items, signals, etc., fails to providethe desired relationship.

As may also be used herein, the terms “processing module”, “processingcircuit”, “processor”, “processing device” and/or “processing unit” maybe a single processing device or a plurality of processing devices. Sucha processing device may be a microprocessor, micro-controller, digitalsignal processor, graphics processing unit, microcomputer, centralprocessing unit, field programmable gate array, programmable logicdevice, state machine, logic circuitry, analog circuitry, digitalcircuitry, and/or any device that manipulates signals (analog and/ordigital) based on hard coding of the circuitry and/or operationalinstructions. The processing module, module, processing circuit, and/orprocessing unit may be, or further include, memory and/or an integratedmemory element, which may be a single memory device, a plurality ofmemory devices, and/or embedded circuitry of another processing module,module, processing circuit, and/or processing unit. Such a memory devicemay be a read-only memory, random access memory, volatile memory,non-volatile memory, static memory, dynamic memory, flash memory, cachememory, and/or any device that stores digital information. Note that ifthe processing module, module, processing circuit, and/or processingunit includes more than one processing device, the processing devicesmay be centrally located (e.g., directly coupled together via a wiredand/or wireless bus structure) or may be distributedly located (e.g.,cloud computing via indirect coupling via a local area network and/or awide area network). Further note that if the processing module, module,processing circuit, and/or processing unit implements one or more of itsfunctions via a state machine, analog circuitry, digital circuitry,and/or logic circuitry, the memory and/or memory element storing thecorresponding operational instructions may be embedded within, orexternal to, the circuitry comprising the state machine, analogcircuitry, digital circuitry, and/or logic circuitry. Still further notethat, the memory element may store, and the processing module, module,processing circuit, and/or processing unit executes, hard coded and/oroperational instructions corresponding to at least some of the stepsand/or functions illustrated in one or more of the Figures and/ordescribed herein. Such a memory device or memory element can be includedin an article of manufacture. While the processing module, module,processing circuit, and/or processing unit device may be a generalpurpose computing device, the execution of the hard coded and/oroperational instructions by the processing module, module, processingcircuit, and/or processing unit configures such a general purposecomputing device as a special purpose computing device to implement thecorresponding steps and/or functions illustrated in one or more of theFigures and/or described herein. In particular, the hard coded and/oroperational instructions by the processing module, module, processingcircuit, and/or processing unit implement acts and algorithms performedby the processing module, module, processing circuit, and/or processingunit. Such acts and algorithms can be identified by name, can beillustrated via flowchart and/or described in words.

One or more embodiments have been described above with the aid of methodsteps illustrating the performance of specified functions andrelationships thereof. The boundaries and sequence of these functionalbuilding blocks and method steps have been arbitrarily defined hereinfor convenience of description. Alternate boundaries and sequences canbe defined so long as the specified functions and relationships areappropriately performed. Any such alternate boundaries or sequences arethus within the scope and spirit of the claims. Further, the boundariesof these functional building blocks have been arbitrarily defined forconvenience of description. Alternate boundaries could be defined aslong as the certain significant functions are appropriately performed.Similarly, flow diagram blocks may also have been arbitrarily definedherein to illustrate certain significant functionality.

To the extent used, the flow diagram block boundaries and sequence couldhave been defined otherwise and still perform the certain significantfunctionality. Such alternate definitions of both functional buildingblocks and flow diagram blocks and sequences are thus within the scopeand spirit of the claims. One of average skill in the art will alsorecognize that the functional building blocks, and other illustrativeblocks, modules and components herein, can be implemented as illustratedor by discrete components, application specific integrated circuits,processors executing appropriate software and the like or anycombination thereof.

In addition, a flow diagram may include a “start” and/or “continue”indication. The “start” and “continue” indications reflect that thesteps presented can optionally be incorporated in or otherwise used inconjunction with other routines. In this context, “start” indicates thebeginning of the first step presented and may be preceded by otheractivities not specifically shown. Further, the “continue” indicationreflects that the steps presented may be performed multiple times and/ormay be succeeded by other activities not specifically shown. Further,while a flow diagram indicates a particular ordering of steps, otherorderings are likewise possible provided that the principles ofcausality are maintained.

The one or more embodiments are used herein to illustrate one or moreaspects, one or more features, one or more concepts, and/or one or moreexamples. A physical embodiment of an apparatus, an article ofmanufacture, a machine, and/or of a process may include one or more ofthe aspects, features, concepts, examples, etc. described with referenceto one or more of the embodiments discussed herein. Further, from figureto figure, the embodiments may incorporate the same or similarly namedfunctions, steps, modules, etc. that may use the same or differentreference numbers and, as such, the functions, steps, modules, etc. maybe the same or similar functions, steps, modules, etc. or differentones.

The term “system” is used in the description of one or more of theembodiments. A system implements one or more functions via a device suchas a processor or other processing device or other hardware that mayinclude or operate in association with a memory that stores operationalinstructions. A system may operate independently and/or in conjunctionwith software and/or firmware. As also used herein, a system may containone or more sub-system, each of which may be one or more systems.

As may further be used herein, a computer readable memory includes oneor more memory elements. A memory element may be a separate memorydevice, multiple memory devices, or a set of memory locations within amemory device. Such a memory device may be a read-only memory, randomaccess memory, volatile memory, non-volatile memory, static memory,dynamic memory, flash memory, cache memory, and/or any device thatstores digital information. The memory device may be in a form a solidstate memory, a hard drive memory, cloud memory, thumb drive, servermemory, computing device memory, and/or other physical medium forstoring digital information.

While particular combinations of various functions and features of theone or more embodiments have been expressly described herein, othercombinations of these features and functions are likewise possible. Thepresent disclosure is not limited by the particular examples disclosedherein and expressly incorporates these other combinations.

What is claimed is:
 1. An intensity transformation augmentation system,comprising: at least one processor; and a memory that stores operationalinstructions that, when executed by the at least one processor, causethe intensity transformation augmentation system to: receive a trainingset of medical scans, wherein the training set of medical scans includesa first subset of medical scans that corresponds to a first anatomicalregion and a second subset of medical scans that corresponds to a secondanatomical region; generate a plurality of sets of augmented images,wherein each set of augmented images in the plurality of sets ofaugmented images is generated by performing a set of intensitytransformation functions on one of the training set of medical scans,wherein each of the set of intensity transformation functions are basedon density properties of corresponding one of a plurality of differentanatomy features present in the training set of medical scans wherein afirst proper subset of the set of intensity transformation functions isapplied to medical scans of the first subset based on a first subset ofthe plurality of different anatomy features present in the firstanatomical region, wherein a second proper subset of the set ofintensity transformation functions is applied to medical scans of thesecond subset based a second subset of the plurality of differentanatomy features present in the second anatomical region, and wherein aset difference between the first proper subset of the set of intensitytransformation functions and the second proper subset of the set ofintensity transformation functions is non-null; generate a computervision model by performing a training step on the plurality of sets ofaugmented images, wherein each augmented image of a set of augmentedimages is assigned same output label data based on a corresponding oneof the training set of medical scans; receive a new medical scan;generate inference data by performing an inference function thatutilizes the computer vision model on the new medical scan; and transmitthe inference data to a client device for display via a display device.2. The intensity transform augmentation system of claim 1, wherein theplurality of different anatomy features include bone, soft tissue, fat,fluid, and air.
 3. The intensity transform augmentation system of claim1, wherein performing one of the set of intensity transformationfunctions on the one of the training set of the medical scans includesadjusting a contrast of image data of the one of the training set ofmedical scans to accentuate the corresponding one of the plurality ofdifferent anatomy features.
 4. The intensity transform augmentationsystem of claim 1, wherein performing one of the set of intensitytransformation functions on the one of the training set of the medicalscans includes adjusting a brightness of image data of the one of thetraining set of medical scans to accentuate the corresponding one of theplurality of different anatomy features.
 5. The intensity transformaugmentation system of claim 1, wherein the training set of medicalscans includes a third subset of medical scans that corresponds to afirst abnormality output label and a fourth subset of medical scans thatcorresponds to a second abnormality output label, wherein a third propersubset of the set of intensity transformation functions is applied tomedical scans of the third subset based on first density properties ofthe first abnormality output label, wherein a fourth proper subset ofthe set of intensity transformation functions is applied to medicalscans of the fourth subset based on second density properties of thesecond abnormality output label, and wherein a set difference betweenthe third proper subset of the set of intensity transformation functionsand the fourth proper subset of the set of intensity transformationfunctions is non-null.
 6. The intensity transform augmentation system ofclaim 1, wherein the training set of medical scans includes a thirdsubset of medical scans that corresponds to a first region of interestoutput label and a fourth subset of medical scans that corresponds to asecond region of interest output label, wherein a third proper subset ofthe set of intensity transformation functions is applied to medicalscans of the third subset based on a first subset of the plurality ofdifferent anatomy features that correspond to the first region ofinterest output label, wherein a fourth proper subset of the set ofintensity transformation functions is applied to medical scans of thefourth subset based on a second subset of the plurality of differentanatomy features that correspond to the second region of interest outputlabel, and wherein a set difference between the third proper subset ofthe set of intensity transformation functions and the fourth propersubset of the set of intensity transformation functions is non-null. 7.The intensity transform augmentation system of claim 1, wherein the setof intensity transformation functions is performed on image data of thetraining set of medical scans to produce the plurality of sets ofaugmented images, and wherein generating the computer vision modelincludes performing the training step on the image data of the trainingset of medical scans and the plurality of sets of augmented images. 8.The intensity transform augmentation system of claim 1, wherein each ofthe set of intensity transformation functions is performed on raw sensordata of one of the training set of medical scans to generate one of acorresponding set of augmented images.
 9. The intensity transformaugmentation system of claim 8, wherein the raw sensor data is in afirst data format and wherein the one of the corresponding set ofaugmented images is in a second data format.
 10. The intensity transformaugmentation system of claim 9, wherein the second data format is a JPEGimage format.
 11. The intensity transform augmentation system of claim8, wherein performing each of the set of intensity transformationfunctions includes calculating each one of a plurality of pixel valuesof the augmented image as a deterministic function of each correspondingone of a plurality of density values of the raw sensor data.
 12. Theintensity transform augmentation system of claim 11, wherein thedeterministic function utilized by at least one of the set of intensitytransformation functions is a non-linear function of density value. 13.The intensity transform augmentation system of claim 12, wherein thedeterministic function utilized by at least one of the set of intensitytransformation functions is in accordance with a cumulative distributionfunction of density value, and wherein the cumulative distributionfunction is in accordance with a normal distribution.
 14. The intensitytransform augmentation system of claim 1, wherein each of the set ofintensity transformation functions is in accordance with a differentdensity window boundary pair.
 15. The intensity transform augmentationsystem of claim 1, wherein the operational instructions, when executedby the at least one processor, further cause the intensitytransformation augmentation system to: receive function parameter datafrom a client device, wherein the function parameter data corresponds touser input in response to a prompt displayed via a user interface on adisplay device that corresponds to the client device; and perform atleast one of the set of intensity transformation functions in accordancewith the function parameter data.
 16. The intensity transformaugmentation system of claim 1, wherein the same output label dataindicates known abnormality data of the corresponding one of thetraining set of medical scans, and wherein the inference data indicatesat least one abnormality detected in the new medical scan.
 17. Theintensity transform augmentation system of claim 1, wherein each medicalscan of the training set of medical scans includes a set of imageslices, and wherein the set of intensity transformation functions isperformed on each of the set of image slices on one of the training setof medical scans to generate a corresponding set of augmented images ofthe plurality of sets of augmented images.
 18. The intensity transformaugmentation system of claim 1, wherein the training set of medicalscans includes a plurality of longitudinal data sets that each include aplurality of medical scans, wherein the set of intensity transformationfunctions is performed on each of the plurality of longitudinal datasets to generate a corresponding longitudinal set of augmented images ofthe plurality of sets of augmented images.
 19. A method for execution byan intensity transform augmentation system, comprising: receiving atraining set of medical scans, wherein the training set of medical scansincludes a first subset of medical scans that corresponds to a firstanatomical region and a second subset of medical scans that correspondsto a second anatomical region; generating a plurality of sets ofaugmented images, wherein each set of augmented images in the pluralityof sets of augmented images is generated by performing a set ofintensity transformation functions on one of the training set of medicalscans, wherein each of the set of intensity transformation functions arebased on density properties of corresponding one of a plurality ofdifferent anatomy features present in the training set of medical scanswherein a first proper subset of the set of intensity transformationfunctions is applied to medical scans of the first subset based on afirst subset of the plurality of different anatomy features present inthe first anatomical region, wherein a second proper subset of the setof intensity transformation functions is applied to medical scans of thesecond subset based a second subset of the plurality of differentanatomy features present in the second anatomical region, and wherein aset difference between the first proper subset of the set of intensitytransformation functions and the second proper subset of the set ofintensity transformation functions is non-null; generating a computervision model by performing a training step on the plurality of sets ofaugmented images, wherein each augmented image of a set of augmentedimages is assigned same output label data based on a corresponding oneof the training set of medical scans; receiving a new medical scan;generating inference data by performing an inference function thatutilizes the computer vision model on the new medical scan; andtransmitting the inference data to a client device for display via adisplay device.
 20. The method of claim 19, wherein performing one ofthe set of intensity transformation functions on the one of the trainingset of the medical scans includes adjusting at least one of: a contrastof image data of the one of the training set of medical scans, or abrightness of image data of the one of the training set of medicalscans, to accentuate the corresponding one of the plurality of differentanatomy features.